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ABSTRACT 


MSYS  is  a system  for  reasoning  with  uncertain  information  and  inexact 
rules  of  inference.  Its  major  application,  to  date,  has  been  to  the 
interpretation  of  visual  features  (such  as  regions)  in  scene  analysis.  In 
this  application,  features  are  assigned  sets  of  possible  interpretations 
with  associated  likelihoods  based  on  local  attributes  (e.g.,  color,  size, 
and  shape).  Interpretations  are  related  by  rules  of  inference  that  adjust 
the  likelihoods  up  or  down  in  accordance  with  the  interpretation  likeli- 
hoods of  related  features.  An  asynchronous  relaxation  process  repeatedly 
applies  the  rules  until  a consistent  set  of  likelihood  values  is  attained. 
At  this  point,  several  alternative  interpretations  still  exist  for  each 
feature.  One  feature  is  chosen  and  the  most  likely  of  its  alternatives 
is  assumed.  The  rules  are  then  used  in  this  more  precise  context  to 
determine  likelihoods  for  the  interpretations  of  remaining  features  by  a 
further  round  of  relaxation.  The  selection  and  relaxation  steps  are  re- 
peated until  all  features  have  been  interpreted. 

Scene  interpretation  typifies  constraint  optimization  problems  involv- 
ing the  assignment  of  values  to  a set  of  mutually  constrained  variables. 

For  an  interesting  class  of  constraints,  MSYS  is  guaranteed  to  find  the 
optimal  solution  with  less  branching  than  conventional  heuristic  search 
methods  . 

MSYS  is  implemented  as  a network  of  asynchronous  parallel  processes. 
The  implementation  provides  an  effective  way  of  using  data  driven  systems 
with  distributed  control  for  optimal  stochastic  search. 
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I INTRODUCTION 


A 


In  scene  analysis,  it  is  frequently  impossible  to  interpret  parts  of 
an  image  taken  out  of  context.  Different  objects  may  have  similar  appear- 
ances, while  objects  belonging  to  the  same  functional  class  can  have 
strikingly  different  appearances  (e.g.,  chairs).  Ambiguous  local  inter- 
pretations must  be  ruled  out  by  using  contextual  constraints  to  achieve  a 
meaningful,  globally  consistent  interpretation  of  the  whole  scene. 

We  use  an  elementary  example  involving  arbitrary  constraints  to  illus- 
trate the  reasoning  entailed  in  scene  interpretation.  A room  scene  is 
manually  partitioned  into  regions,  as  shown  in  Figure  1 . The  labels  in 
the  figure  indicate  the  locally  possible,  interpretations  of  each  region. 

These  interpretations  are  obtained  by  matching  region  attributes,  such 

* 

as  height  and  orientation,  against  local  constraints  given  in  Figure  2. 

Region  DR,  for  example,  must  be  either  DOOR  or  WALL,  since  these  are  the 
only  vertically  oriented  bjects  that  can  extend  both  below  and  above  the 
allowed  height  ranges  of  other  objects,  such  as  PICTURES,  CHAIRBACKS,  and 
WASTEBASKETS.  In  this  example,  horizontally  oriented  regions  all  received 
unique  interpretations  determined  by  their  height,  but  all  vertical  regions 
received  at  least  two  possible  interpretations  [DOOR,  WALL,  and  when  con- 
sistent with  height  extremes.  WASTEBASKET,  CHAIRBACK,  or  PICTURE].  It  is 
assumed  that  regions  do  not  span  more  than  one  object  . 

When  the  interpretation  of  a region  cannot  be  uniquely  determined 
from  local  attributes,  it  must  be  deduced  from  global  relationships  such 
as  those  in  Figure  2.  Deduction  might  proceed  as  follows: 

* 

The  scene  analysis  experiments  reported  herein  were  performed  using 
coordinated  arrays  of  color,  intensity,  and  range  data.  The  range  data 
simulated  the  output  of  a developmental  time-of-f light  laser  range  finder, 
whose  current  accuracy  is  about  an  inch  in  ten  feet.  Region  height  and 
orientation  were  obtained  from  the  range  data  using  transformations  de- 
scribed in  Reference  1.  The  local  interpretations  shown  in  Figure  1 were 
obtained  automatically  using  the  measured  height  and  orientation  of  the 
regions . 
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INTERPRETATION 
1.  FLOOR 


2.  DOOR 


LOCAL  CONSTRAINTS 

(HEIGHT  FLOOR)  < 0.1  feel 
(ORIENTATION  FLOOR)  ~ HORIZONTAL 

0 < (HEIGHT  DOOR)  < 7 feet 
(ORIENTATION  DOOR)  ~ VERTICAL 


RELATIONAL  CONSTRAINTS 
(HOMOGENEOUS  FLOOR)* 


(HOMOGENEOUS  DOOR) 

(NOT  (ADJACENT  DOOR  PICTURE)) 
(ROOMPARTITION  DOOR)** 


3.  WALL 


0 < (HEIGHT  WALL)  < 8 feet 
(ORIENTATION  WALL)  ~ VERTICAL 


(HOMOGENEOUS  WALL) 
(ROOMPARTITION  WALL) 


4.  PICTURE  3 < (HEIGHT  PICTURE)  « 5.5  feet 

(ORIENTATION  PICTURE)  ~ VERTICAL 

5.  CHAIRBACK  1.5  < (HEIGHT  CHAIRBACK)  < 3 feet 

(ORIENTATION  CHAIRBACK)  ~ VERTICAL 


(NOT  (ADJACENT  PICTURE  DOOR)) 


6.  CHAIRSEAT 


1 < (HEIGHT  CHAIRSEAT)  < 2 feet 
(ORIENTATION  CHAIRSEAT)  ~ HORIZONTAL 


7.  TABLETOP  2 < (HEIGHT  TABLETOP)  < 3 feet 

(ORIENTATION  TABLETOP)  ~ HORIZONTAL 


8.  WASTEBASKET  0 < (HEIGHT  WASTEBASKET)  < 1.5  feet 


•The  homogeneity  constraint  requires  that  all  regions  labeled  with  the  constrained  interpretation,  in  this  case  Floor, 
have  approximate!'  the  same  brightness  arid  color.  No  particular  brightness  or  color  is,  however,  required. 

••The  roompar  tition  constraint  requires  that  all  regions  labeled  with  the  constrained  interpretation  (Door  or  Wall) 
have  appro  .imately  the  same  brightness  found  along  the  top  edge  of  the  image  vertically  above  that  region's  center 
of  mass.  This  constraint  is  based  on  the  assumptions  that  Door  and  Wall  are  homogeneous  (as  defined  above),  and  that 
in  a standard  eye  level  view  of  a normal  sized  room.  Door  and  Wall  will  extend  beyond  the  upper  border  of  the 
image.  This  constraint  is  a special  case  of  homogeneity  and  would  not  be  required  for  analyzing  exhaustively  partitioned 
scenes. 


FIGURE  2 SEMANTIC  CONSTRAINTS  FOR  ROOM  SCENE  DEPICTED  IN  FIGURE  1 
(PARTIAL  LIST) 
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• Regions  PIC,  WBSKT,  and  CBACK  cannot  be  WALL  or  DOOR,  because 
their  brightnesses  are  much  less  than  that  along  the  top  edge 

of  the  image  vertically  above  them,  which  violates  the  constraint 
ROOMPARTITION . Consequently,  region  PIC  must  be  the  PICTURE, 

WBSKT  must  be  WASTEBASKET,  and  CBACK  must  be  CHAIRBACK. 

• Region  LWALL  and  RWALL  must  then  be  WALL,  since  they  are  adjacent 
to  region  PIC,  and  DOOR  cannot  be  adjacent  to  PICTURE. 

• Region  DR  cannot  be  WALL  because  all  regions  labeled  WALL  are 
required  to  have  the  same  brightness.  Therefore,  region  DR  must 
be  DOOR. 

Scene  interpretation,  as  illustrated  by  the  above  example,  is  an 
attempt  to  explain  observed  sensory  data  in  terms  of  prior  knowledge  about 
the  depicted  domain.  The  explanation  can  entail  many  types  and  levels  of 
knowledge,  some  of  which  may  be  probablistic  or  inconsistent.  It  must 
also  allow  for  the  likelihood  that  the  data  is  noisy.  For  these  reasons, 
scene  interpretation  is  not  a purely  deductive  problem  with  a unique  cor- 
rect solution;  it  is  a problem  that  requires  a search  for  the  best  or 
optimum  explanation.  However,  the  quantities  of  data  and  knowledge  that 
are  involved  appear  to  rule  out  the  use  of  conventional  search  techniques. 

This  paper  describes  a working  scene  interpretation  program,  called 
MSYS,  in  which  knowledge  sources  compete  and  cooperate  until  a consistent 
explanation  of  the  sce^e  emerges  by  consensus.  The  consensus  is  achieved 
by  a network  of  processes  (representing  independent  knowledge  sources) 
that  communicate  via  shared  global  variables.  Each  process  attempts  to 
explain  a fragment  of  the  data  (a  region  or  a few  regions  in  a segmented 
scene)  in  terms  of  its  own  limited  knowledge.  The  confidence  of  an  ex- 
planation is  communicated  to  other  processes  attempting  to  explain  over- 
lapping fragments,  and  may  cause  them  to  reevaluate  their  own  hypothesis. 
The  confidence  adjustment  cycle  continues  until  equilibrium  is  achieved. 

The  equilibrium  confidence  values  establish  a preference  ordering  for  the 
alternative  interpretations  of  each  fragment,  which  is  used  to  guide  a 
heuristic  search  toward  the  best  solution.  We  conjecture  that,  with 
enough  knowledge  (i.e.,  constraints),  the  equilibrium  state  will  correspond 
directly  to  a solution  (where  one  interpretation  for  each  fragment  is  by 
far  the  best).  A competent  knowledge-based  vision  system  would  thus  never 
actually  need  to  resort  to  search. 
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function.  In  Section  III,  we  describe  MSYS,  which  is  an  efficient  serial 

* 

implementation  of  the  M algorithm  using  (simulated)  asynchronous  parallel 
processes.  In  Section  IV  we  contrast  MSYS  with  previous  work  on  scene 
interpretation  and  constraint  optimization  and  suggest  applications  of  our 
work  in  both  areas. 

The  work  described  in  this  report  was  motivated  by  the  work  of  Duda 
at  SRI,  Barrow  and  Turner  at  Edinburgh,  and  Yakimovsky  and  Feldman  at 
Stanford.  Duda20  was  concerned  with  assigning  interpretations  to  regions 
in  a previously  segmented  scene.  The  interpretation  process  involved  a 
tree  search  to  determine  the  set  of  region  interpretations  having  the 
highest  joint  likelihood . The  inefficiencies  of  tree  search  limited  this 
approach  to  simple  scenes  with  few  regions  and  few  interpretations. 

Barrow  and  Turner3  developed  an  elegant  generalization  of  Waltz's 
filtering  algorithm,  that  dramatically  reduced  the  amount  of  search 
required  to  solve  constraint  satisfaction  problems.  Yakimovsky  and 
Feldman2  showed  how  segmentation  and  interpretation  could  be  integrated 
by  using  the  likelihoods  of  region  interpretations  to  guide  region  merging. 
In  this  paper,  we  describe  the  combination  of  these  ideas  into  a system 
that  can  efficiently  determine  optimal  region  interpretations  in  a 
segmented  image.  This  work  was  begun  in  1973;  preliminary  results  were 
previously  reported  in  Reference  4. 


II  THE  M ALGORITHM 

* 

M is  a heuristic  search  algorithm  intended  for  multivariate  optimi- 
zation problems  involving  interacting  nonlinear  constraints.  Problems 
are  posed  by  providing  (1)  a set  of  possible  assignments  for  each  variable, 
with  associated  a priori  likelihoods  and  (2)  a set  of  constraints  that 
determine  the  a posteriori  likelihood  of  any  variable  assignment  for  a 
given  instantiation  of  the  remaining  variables. 

A solution  of  the  problem — also  called  a terminal  state — is  any 
complete  instantiation  of  the  variables.  A partial  solution  is  a nonter- 
minal state  in  which  at  least  one  variable  still  has  a set  of  possible 

assignments.  The  objective  is  to  find  the  solution  in  which  the  combined 

* 

a posteriori  likelihoods  of  the  instantiated  assignments  is  greatest.  M 

uses  a relaxation  method  to  solve  simultaneously  the  set  of  constraint 

equations  that  determine  the  overall  merit  of  a solution  and  also  to  bound 

the  potential  merits  of  partial  solutions.  These  latter  estimates  are 

* 

used  to  guide  a conventional  A search  algorithm  toward  the  optimal  solu- 
tion . 

* 

The  M algorithm  was  initially  formulated  specifically  for  scene 
interpretation  and  is  described  here  in  those  terms.  However,  it  appears 
to  have  broad  applicability  as  a general  search  algorithm,  as  is  suggested 
later  in  this  report. 

The  scene  interpretation  problem  can  be  defined  more  formally  as 
follows:  Given  a set  of  regions  and  corresponding  region  attributes,  a 

generic  set  of  possible  region  interpretations  and  a set  of  constraints 
on  the  generic  interpretations  determine  the  assignment  of  interpretations 
to  regions  that  best  satisfies  the  constraints  (i.e.,  that  assignment  for 
which  the  combined  a posteriori  likelihoods  of  the  interpretations  is 
greatest)  . The  search  for  this  optimal  solution  starts  from  an  initial 
state  in  which  all  regions  have  sets  of  possible  interpretations.  It 
then  proceeds  through  a series  of  partially  instantiated  states  to  terminal 
states  in  which  every  region  has  been  instantiated  to  a unique  interpre- 
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tation.  Region  interpretations  are  instantiated  by  pinning  the  likeli- 
hoods of  alternative  interpretations  of  the  region  to  zero. 

A . State  Evaluation 

Each  interpretation  in  a terminal  state  has  a likelihood  that  is  a 
heuristic  function  of  the  local  region  attributes  and  the  likelihoods  of 
the  interpretations  assigned  to  other  regions.  This  likelihood  tells 
how  well  the  semantic  constraints  on  the  interpretation  are  satisfied  in 
! , the  current  state.  The  precise  form  of  a likelihood  function  need  not 

concern  us  at  this  time,  except  to  note  that  the  likelihood  associated 
with  any  particular  region  interpretation  is,  in  general,  a nonlinear 
combination  of  the  likelihoods  associated  with  all  the  other  region  inter- 
pretations. Hence,  the  determination  of  likelihoods  in  a terminal  state 
may  involve  the  simultaneous  solution  of  a set  of  nonlinear  equations.  A 
relaxation  method,  described  more  fully  in  Section  III.C,  is  used  to  ob- 
tain a consistent  set  of  likelihood  values. 

A terminal  state  will  be  scored  by  summing  the  interpretation  likeli- 
hoods over  all  regions.  The  optimization  objective,  then,  is  to  find  the 
highest  scoring  terminal  state,  hopefully  without  exhaustive  enumeration. 

To  avoid  enumeration,  a heuristic  search  is  desired.  A heuristic 
search  estimates  the  best  terminal  scores  that  could  be  ultimately  achieved 
by  further  instantiating  a given  partially  instantiated  state.  Search 
then  proceeds  by  developing  the  state  with  the  best  potential  terminal 
score.  The  best  terminal  score  attainable  from  a given  partially  instan- 
tiated state  is  estimated  by  calculating,  for  each  region  interpretation 

I 

remaining  in  that  state,  an  upper  limit  on  the.  a posteriori  likelihood  of 

■ " 

that  interpretation  in  any  terminal  state.  The  likelihoods  of  the  most 

I * 

likely  interpretation  of  all  regions  are  then  summed  to  provide  an  upper- 
r bound  on  the  best  terminal  score  that  could  be  obtained . 


likelihood  under  consideration,  if  only  that  one  constraint  applied.  An 
upperbound  on  the  likelihood  of  the  current  interpretation  is  then  computed 
on  the  basis  of  these  individually  optimized  constraints.  (For  example, 
the  likelihood  estimate  could  be  based  on  the  strength  of  the  least  satis- 
fied constraint.) 

The  likelihood  limits  of  all  interpretations  are,  of  course,  inter- 
dependent and  should  thus  be  computed  simultaneously  to  obtain  a tighter 
bound  on  the  overall  state  score.  In  other  words,  the  likelihood  limit 
of  each  interpretation  should  be  computed  assuming  all  other  interpretation 
likelihoods  are  at  their  upper  limits . A consistent  set  of  likelihood  limits 
can  be  obtained  with  the  same  relaxation  method  used  earlier  to  obtain 
consistent  likelihoods  in  terminal  states  by  suitably  modifying  the  like- 
lihood functions. 

Not  much  can  be  said  regarding  either  convergence  of  the  relaxation 
process  or  the  quality  of  the  resulting  likelihood  estimates  without 
knowing  more  about  the  functions  that  compute  likelihood.  Two  important 
classes  of  likelihood  functions  are  those  that  increase  the  likelihood  of 
a constrained  interpretation  when  the  likelihood  of  compatible  interpre- 
tations elsewhere  in  the  image  increase,  and  those  that  decrease  likelihoods 
when  the  likelihood  of  incompatible  interpretations  increase.  If  we  restrict 
ourselves  to  the  former,  then  the  partial  derivatives  of  any  interpretation 
likelihood,  taken  with  respect  to  the  likelihood  of  any  other  interpretation 
likelihood,  will  be  positive.  The  positive  partial  derivatives  guarantee 
that  the  relaxation  process  will  be  nonoscillatory : all  likelihoods  will 

monotonical ly  increase  (or  decrease)  until  either  a stable  state  is  achieved, 
or  the  limit  0 (or  1)  is  reached.  Consequently,  the  process  is  guaranteed 
to  converge  providing  a consistent  solution  exists  in  that  range.  When 
a region  is  instantiated  to  some  interpretation,  the  likelihoods  of  alter- 
native interpretations  of  that  region  are  reduced  to  zero.  This  can  cause 
monotonic  reductions  in  the  likelihood  estimates  of  other  region  interpre- 
tation, but  no  increases.  Thus,  the  estimated  score  of  a partially  instan- 
tiated state  is  a strict  upperbound  on  the  score  of  any  terminal  state  that 
can  be  reached  along  that  branch  of  the  search  tree. 
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B . Heuris tic  Search 

The  above  results  suggest  a heuristic  search  strategy  patterned  after 

* 

the  A algorithm  of  Hart,  Nilsson  and  Raphael.0  The  search  proceeds,  at 
each  stage,  by  restoring  the  highest  scoring  partially  instantiated  state  and 
then  instantiating  the  highest  likelihood  interpretation  of  an  uninstantiated 
region  in  a new  copy  of  that  state. 


Following  the  instantiation,  the  relaxation  process  is  repeated  twice 
first,  to  update  the  estimated  likelihoods  and  potential  score  of  the  new 
instantiated  state,  and  second,  to  update  estimates  in  the  original  state  with 
the  instantiated  interpretation  removed  as  a possibility.  (Effectively,  the 
search  is  split  into  2 disjoint  branches.)  Both  states  are  added  to  a 
prioritized  list  of  open  states.  Search  then  continues  in  the  highest  scoring 
state,  terminating  when  the  best  state,  is  also  a terminal  state.  In  Appendix 
A it  is  proved  that  this  algorithm,  with  the  stated  restrictions  on  the  formal 
likelihood  functions  will  terminate  with  the  optimal  set  of  interpretations 
for  those  constraints . 

* 

The  complete  M algorithm,  as  actually  implemented,  is  summarized  in 
Figure  3.  The  heart  of  the  algorithm  is  the  use  of  relaxation  methods  in 
Steps  0 and  4 to  estimate  consistent  likelihood  limits  of  alternative 
interpretations  of  regions.  These  estimates  are  then  used  to  improve  the 
order  of  state  selection  and  instantiation  at  Steps  1 and  2,  respectively. 

Step  5 introduces  an  additional  constraint  on  the  solution,  namely,  that 
all  regions  have  at  least  one  reasonably  likely  interpretation;  any  state 
in  which  all  possible  interpretations  of  a region  receive  unacceptably  low 
likelihoods  is  abandoned.  Step  6 allows  the  algorithm  to  terminate  early, 
whenever  all  regions  in  the  highest  scoring  state  have  one  clearly  dominant 
interpretation  (i.e.,  an  interpretation  at  least  ten  times  as  likely  as 
any  alternative),  whether  or  not  the  dominant  interpretations  have  been 
formally  instantiated. 

* 

While  the  basic  search  algorithm  used  by  M is  inherently  serial,  the 
relaxation  process  that  guides  the  search  is  conceptually  parallel.  All 
possible  interpretations  could,  in  principle,  be  represented  by  independent 
processes  that  interact  to  achieve  equilibrium  likelihoods.  Considering 
the  highly  constrained  nature  of  most  scene  interpretation  problems,  one 
might  hope  that  the  equilibrium  likelihoods  of  "correct"  region  interpre- 
tations would  dominate  those  of  "incorrect"  alternative  interpretations, 
prior  to  any  instantiation,  and  that  this  dominance  w'ould  be  further  en- 
hanced in  the  equilibrium  states  resulting  from  each  subsequent  correct 
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(0)  Establish  consistent  likelihood  limits  for  all  interpretations  based  on  the  a priori  likelihoods  and  constraints. 
Record  all  non-uniquely  instantiated  regions  (on  IQUEUE)  and  save  the  resultant  state  (on  SQUEUE)*. 

(1)  Reinstate  the  current  globally  best  state  (from  SQUEUE). 

(2)  Select  a region  interpretation  for  instantiation  (from  IQUEUE). 

(3)  Generate  branches  corresponding  to  acceptance  and  rejection  of  that  instantiation  hypothesis,  setting  up  a new 
state  for  each. 

(4)  In  each  of  the  new  states  reevaluate  all  interpretation  likelihoods  affected  by  acceptance  (or  rejection)  of  the 
hypothesized  interpretation;  pursue  the  consequences  of  all  reevaluations  as  far  as  possible  short  of  further 
hypothesizing. 

(5)  Evaluate  the  global  score  of  each  state  by  summing  the  likelihoods  of  the  best  interpretation  for  each  region.  Any 
state  in  which  all  possible  interpretations  of  some  region  are  deleted  (or  receive  a very  low  likelihood)  is  assigned 

a zero  global  likelihood. 

(6)  if  ait  regions  in  either  state  are  assigned  unique  interpretations,  terminate  and  return  that  state  as  the  best  scene 
interpretation.  If  all  regions  in  both  states  are  uniquely  interpreted,  return  the  state  with  the  highest  score. 

(7)  Update  the  IQUEUE  associated  with  each  state  and  save  both  states  (on  SQUEUE)  with  priority  determined  by  their 
respective  scores. 

(8)  Go  to  (1). 

•The  roles  of  SQUEUE  and  IQUEUE  are  explained  in  section  I1I-D. 

FIGURE  3 BEST  FIRST  SEARCH  ALGORITHM 
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instantiation.  This  expectation  has  been  substantially  confirmed  in 
experimentation  with  a variety  of  likelihood  functions,  including  many 
that  did  not  satisfy  the  restrictions  required  for  formal  optimality. 
Consequently,  search  proceeds  toward  the  correct  solution  with  little  or 
no  backup  and  instantiation  becomes  virtually  a serial  readout  of  a paral- 
lel search. 

C . An  Example 

The  M algorithm  is  illustrated  with  the  same  regions,  interpretations, 
and  constraints  used  in  the  introductory  example  (Figures  1 and  2),  but 
with  likelihoods  attached  to  the  interpretations.  The  a priori  interpre- 
tation likelihoods  for  the  example  are  shown  in  Figure  4.  These  values 
are  based  on  the  relative  areas  occupied  by  each  of  the  alternative  inter- 
pretations of  a region  in  several  training  scenes . (The  a priori  likelihood 
of  a region  interpretation  was  computed  by  dividing  the  amount  of  area  with 
that  interpretation  in  the  training  images,  by  the  sum  of  the  corresponding 
areas  over  all  the  possible  interpretations  of  that  region.  WALL  was  thus 
always  a more  likely  a priori  interpretation  than  DOOR  for  vertical  regions.) 

The  first  step  of  analysis  entails  the  estimation  of  a consistent 
likelihood  limit  for  each  locally  possible  region  interpretation,  based 
both  on  the  a priori  (local)  likelihood  of  that  interpretation  and  on  the 
likelihood  limits  estimated  for  other  semantically  constrained  interpre- 
tations in  the  scene.  The  likelihood  limit  of  each  interpretation,  or  just 
likelihood  for  short,  is  computed  by  a function,  hereafter  known  as  a like- 
lihood function.  Likelihood  functions  typically  consist  of  a combination 
(e.g.,  a product)  of  terms — one  for  each  constraint  applicable  to  that 
interpretation.  Each  term  raises  or  lowers  the  likelihood  of  the  inter- 
pretation, depending  on  the  type  of  constraint  and  the  likelihoods  of  the 
region  interpretations  that  satisfy  the  constraint  in  the  current  scene. 

(In  Section  III.B,  we  discuss  in  detail  the  computation  of  likelihoods.) 

Two  basic  relational  constraints  are  used  in  our  present  example: 
adjacency  and  homogeneity.  Both  constraints  reduce  the  likelihood  of 
constrained  interpretations  due  to  the  presence  of  incompatible  interpre- 
tations elsewhere  in  the  image. 
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MANUALLY  PARTITIONED  ROOM  SCENE  WITH  A PRIORI  INTERPRETATION 
LIKELIHOODS  BASED  ON  HEIGHT  AND  SURFACE  ORIENTATION 


FIGURE  4 


The  relational  constraint  (NOT* (ADJACENT  PICTURE  DOOR))  on  Pictures 
(see  Figure  2)  introduces  a term  in  the  likelihood  function  of  region 
interpretation  (PICTURE  PIC),  reducing  the  likelihood  that  region  PIC  is 
PICTURE  by  an  amount  proportional  to  the  likelihood  that  adjacent  regions 
LWALL  and  RWALL  are  thought  to  be  doors.  Conversely,  terms  are  included 
in  the  likelihood  functions  of  the  interpretations  (DOOR  LWALL)  and  (DOOR 
RWALL),  reducing  their  likelihoods  proportional  to  the  likelihood  that 
region  PIC  is  PICTURE.  A loose  definition  of  adjacency  has  been  adopted 
so  this  constraint  could  be  used  in  a partially  segmented  scene.  (Two 
regions  are  adjacent  if  the  line  connecting  their  centers  does  not  pass 
through  a third  region.) 

Homogeneity  constraints  require  that  all  regions  with  a specified 
interpretation  have  approximately  the  same  brightness.  The  likelihood  of 
a constrained  interpretation  is  therefore  reduced  in  proportion  to  the 
maximum  likelihood  that  any  nonhomogeneous  region  also  has  that  interpre- 
tation. In  Figure  4,  for  example,  the  likelihood  that  a light  colored 
region  such  as  LWALL  or  RWALL  is  WALL  must  be  reduced  in  proportion  to 
the  likelihood  that  any  dark  colored  region,  such  as  DR,  PIC,  or  CBACK, 
is  WALL,  and  vice  versa.  It  does  not  matter  whether  WALL  is  light  or 
dark — only  that  dark-  and  light-colored  regions  cannot  with  high  likeli- 
hood simultaneously  be  interpreted  as  WALL. 

ROOMPARTITION,  as  described  above,  is  a special  case  of  the  homogenei ty 
constraint.  The  brightnesses  of  regions  admitting  the  interpretations 
WALL  or  DOOR  (i.e.,  surfaces  that  "partition  rooms")  are  required  to  be 
similar  to  the  brightness  at  the  top  of  the  image  vertically  above  the 
regions  center  of  mass.  Region  interpretations  that  fail  this  test  are 
rejected  by  reducing  their  a pri  ri  likelihoods  to  zero.  The  likelihoods 
of  interpretations  that  pass  are  unaffected.  This  constraint  had  the  effect 
of  eliminating  DOOR  and  WALL  as  possible  interpretations  of  vertically 
oriented  regions  with  inadequate  vertical  extent,  specifically  from  the 
regions  CBACK,  PIC,  and  WBSKT . 

The  interpretation  likelihoods  resulting  from  an  initial  relaxation 
of  the  likelihood  functions  sketched  above  are  shown  in  Figure  5.  In 
that  process,  DOOR  and  WALL  were  eliminated  by  the  constraint  ROOMPARTITION, 
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as  possible  interpretations  of  regions  CBACK,  PIC,  and  WBSKT,  leaving 
region  PIC  with  the  unique  interpretation  of  PICTURE.  The  latter  result 
depressed  the  likelihood  that  adjacent  regions  LWALL  or  RWALL  were  DOOR, 
which  in  turn  enhanced  the  likelihood  that  the  dark  region  DR  was  a door. 
Significantly,  the  correct  interpretation  of  every  region  has  acquired  a 
likelihood  higher  than  that  of  any  alternative  interpretation  for  the 
region . 

The  final  stage  of  analysis  involves  searching  for  a set  of  unique 
interpretations  with  the  highest  joint  likelihood.  The  only  remaining 
ambiguity  involves  the  interpretation  of  regions  DR,  LWALL,  and  RWALL,  all 
of  which  still  admit  both  DOOR  and  WALL  as  possibilities.  Homogeneity 
constraints  force  LWALL  and  RWALL,  both  light-colored  regions,  to  take 
the  same  interpretation  (either  WALL  or  DOOR)  and  DR,  a dark-colored 
region,  to  take  the  opposite  interpretation.  This  basic  ambiguity  is 
resolved  by  the  adjacency  constraint  on  pictures,  which  leads  to  a contra- 
diction when  either  LWALL  or  RWALL  is  instantiated  to  DOOR. 

The  search  proceeded  without  need  for  backup,  the  relative  likelihoods 
of  correct  interpretations  increasing  monotonically  with  each  successive 
correct  instantiation.  The  final  interpretation  likelihoods  for  the  re- 
gions in  Figure  4 are  presented  in  Figure  6.  A detailed  trace  of  the 
reasoning  showing  all  instantiations  and  resulting  reevaluations  appears 
in  Appendix  B. 
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FIGURE  6 FINAL  EQUILIBRIUM  LIKELIHOODS  FOLLOWING  SEARCH 
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III  MSYS— AN  I MPRLE MENTATION  OF  M FOR  SCENE  INTERPRETATION 

MSYS  is  an  operational  system  coded  in  INTERLISP  that  performs  scene 

* 

interpretation  using  the  M algorithm.  The  system  consists  of  four  major 
components : 

• A facility  for  defining  regions  in  a scene  and  for  measuring 
their  pictorial  attributes. 

• An  initialization  procedure  that  compiles,  for  each  possible 
region  interpretation,  a function  that  estimates  the  maximum 
likelihood  of  that  interpretation,  based  on  the  maximum  estimated 
likelihoods  of  the  other  region  interpretations. 

• A relaxation  mechanism  for  determining  consistent  likelihood 
estimates  simultaneously  for  all  the  possible  region  interpre- 
tations . 

• A mechanism  for  performing  a backtrack  search. 

A.  Region  Definition  and  Description 

Region  definition  and  attribute  measurement  are  done  in  MSYS  by  a 
previously  developed  interactive  scene  interpretation  system  known  as 
ISIS.1’28  Regions  can  be  defined  in  ISIS  manually,  by  outlining  them  on 
a display  with  a cursor;  semiautomatically , by  providing  a crude  outline 
which  the  system  then  refines;  or  fully  automatically,  by  calling  a region 
growing  program  similar  to  that  employed  by  Yakimovsky.2 

A variety  of  INTERLISP  functions  are  available  for  assessing  the 
attributes  of  defined  regions.  These  attributes  include  statistics  on 
brightness,  hue,  and  saturation,  as  well  as  height  and  orientation  when 
range  data  is  available.  Functions  also  exist  for  accessing  the  polygonal 
boundaries  of  regions  in  both  (2-D)  image  and  (3-D)  world  coordinates  . 

The  interactive  features  of  ISIS  proved  useful  in  developing  classification 
criteria  for  assigning  local  interpretations,  and  in  devising  procedures 
for  testing  spatial  relations  between  regions. 

B . Compilation  of  Likelihood  Functions 

Before  describing  the  compilation  of  likelihood  functions,  we  digress 
briefly  to  describe  the  structure  of  these  functions  and  their  numerical 
evaluation . 
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1.  Structure  of  Likelihood  Functions 

The  likelihood  of  a region  interpretation  has  been  described  as 
a heuristic  function  of  local  region  attributes,  region  relations,  and 
the  likelihoods  of  other  region  interpretations.  A prototypical  likeli- 
hood function  is  illustrated  in  Figure  7.  Here  (CHAIRSEAT  R3),  the 
likelihood  that  Region  R3  is  CHAIRSEAT  is  represented  as  a conjunction  of 
three  independent  terms.  The  first  term  is  the  a priori  likelihood  (0.8) 
that  Region  R3  is  CHAIRSEAT,  based  on  the  attributes  of  that  region,  such 
as  height  and  surface  orientation.  The  other  two  terms  express,  respectively, 
the  degree  to  which  each  of  the  generic  constraints  on  the  interpretation 
CHAIRSEAT— namely  (ABOVE  CHAIRSEAT  CHAIRLEG)  and  (ABOVE  CHAIRBACK  CHAIR- 
SEAT) — are  satisfied  by  the  interpretation  possibilities  of  the  other  regions. 

The  likelihood  expression  in  Figure  7 can  be  interpreted  in  the 
same  way  as  a LISP  function;  each  subexpression  enclosed  in  parentheses 
is  a function  returning  a real  value,  which  is  then  used  in  evaluating  the 
superexpression  in  which  the  subexpression  appears.  Terms  representing 
region  interpretations,  such  as  (CHAIRLEG  R2),  evaluate  to  the  the  current 
likelihood  of  that  interpretation,  while  terms  representing  region  relations, 
e.g.,  (ABOVE  R4  R3)  express  the  degree  to  which  two  regions  satisfy  the 
specified  relation,  based  on  the  relative  coordinates  of  their  respective 
boundary  extremes.  Both  likelihoods  and  relations  are  defined  the  range 
(0,1).  The  functions  AND*  and  OR*  take  real  valued  arguments  on  the  range 
(0,1)  and  return  values  in  the  same  interval.  They  should  thus  be  consid- 
ered as  general  functions  for  combining  evidence  rather  than  as  conventional 
logical  conjunctions  and  disjunctions.  The  nature  of  these  evidence  com- 
bining functions  will  be  discussed  in  Section  III.B.2.  Functional  definitions 
for  some  common  region  relations  are  given  in  Appendix  C. 

Terms  expressing  the  satisfaction  of  a relational  constraint  such 
as  (ABOVE  CHAIRBACK  CHAIRSEAT)  follow  a standard  format . Each  constraint 
is  supported  (i.e.,  satisfied)  by  a disjunction,  OR*,  of  all  the  potential 
ways  it  can  be  satisfied  in  the  image.  A constraint  is  potentially  satis- 
fied by  a region  admitting  the  required  interpretation  (e.g.,  CHAIRBACK) 
and  having  the  specified  spatial  relationship  with  the  constrained  region 
(e.g.,  ABOVE).  The  region  interpretations  (R4  CHAIRBACK)  and  (R5  CHAIRBACK), 
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POSSIBLE  REGION  INTERPRETATIONS 


REGION 

POSSIBLE  INTERPRETATIONS 
AND  A PRIORI  LIKELIHOOD 

MSVS 

REPRESENTATION 

R1 

CHAIRLEG  0.8 

TABLELEG  0.2 

(CHAIRLEG  R1 1 
(TABLELEG  R1) 

R2 

CHAIRLEG  0.7 

TABLELEG  0.3 
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R3 
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(WALL  R4I 

R5 
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(CHAIRBACK  R5) 
(WALL  R5) 

CONSTRAINTS  ON  CHAIRSEAT 

(ABOVE  CHAIRBACK  CHAIRSEAT) 

(ABOVE  CHAIRSEAT  CHAIRLEG) 

LIKELIHOOD  PROCEDURE  FOR  (CHAIRSEAT  R3) 
[AND*  0.8 
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J 2 2 

(AND*  (ABOVE  R,  R.)  (CHAIRLEG  R.))l 

> 3 1 i 

•J  (OR*  (AND*  (ABOVE  R4  R3)  (CHAIRBACK  R^)) 


for  instance,  provide  potential  satisfaction  of  the  constraint  (ABOVE 
CHA1RBACK  CHAIRSEAT)  in  Figure  7.  The  degree  of  satisfaction  provided  by 
each  potential  supporting  region  interpretation  is  represented  by  the 
conjunction  (AND*)  of  the  interpretation  likelihood  and  the  degree  to  which 
the  region  satisfies  the  stipulated  relation  with  the  constrained  region. 

For  instance,  (AND*  (ABOVE  R4  R3)  (CI1AIRBACK  R4))  expresses  the  degree  to 
which  the  region  interpretation  (CHAIRBACK  R4)  satisfies  the  constraint 
(ABOVE  CHAIRBACK  CHAIRSEAT)  imposed  on  region  interpretation  (CHAIRSEAT  R3) . 

Negated  relations  such  as  (NOT  (ADJACENT  PICTURE  DOOR))  are 
represented  by  first  forming  the  disjunction  of  conjuncts  that  expresses 
support  for  the  basic  constituent  relation,  in  this  case  (ADJACENT  PICTURE 
DOOR) . The  resulting  disjunction  is  then  embedded  in  a negated  clause  of 
the  form  (NOT*  (OR*  - -)).  In  a likelihood  function,  this  term  has  the 
desired  effect  of  penalizing  an  interpretation  to  the  extent  that  the  for- 
bidden relation  is  satisfied  . 

Constraints  that  do  not  fit  the  format  of  binary  relations  are 
represented  within  a likelihood  function  by  support  terms  that  are  arbi- 
trary functions  of  region  attributes,  relations,  and  interpretation  like- 
lihoods. For  example,  the  support  term  representing  the  room  scene  constraint 
(HOMOGENEOUS  DOOR)  ensured  that  two  regions  of  different  brightness  could 
not  be  simultaneously  interpreted  as  DOOR  with  high  likelihood . 

The  procedures  in  Figure  8 are  the  likelihood  functions  that 
were  used  in  the  room  scene  example.  The  listing  omits  functions  of 
unconstrained  interpretations,  such  as  FLOOR,  whose  likelihoods  always 
remain  at  their  a priori  values. 

The  likelihood  function  for  region  interpretation  (DOOR  LWALL) , 
line  57  in  Figure  8,  illustrates  both  a negated  relation  and  a nonrelational 
constraint.  This  likelihood  function  contains  three  terms,  the  first  being 
the  a priori  likelihood  0.227.  The  second  term,  (NOT*  (OR*  (DOOR  DR) 

(DOOR  PIC)  (DOOR  CBACK)  (DOOR  WBSKT) ) ) , expresses  support  for  the  nonstandard 
constraint  (HOMOGENEOUS  DOOR).  This  term  is  the  negation  of  a disjunction 
containing  all  region  interpretations  in  which  the  interpretation  DOOR 
is  paired  with  a region  whose  brightness  differs  from  that  of  the  constrained 
region  (LWALL)  by  more  than  10%.  Its  effect  is  to  reduce  the  likelihood 
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VALHf:  .123 

PROCEDURE: 


PEI  AT  IVES: 


(AND*  .227  (AND*  (NOT*  (OP*  (DOOR  OR) 

(DOOR  PIC) 

(DOOR  CRACK) 
(DOOR  WRSKT ) ) ) 
(NOT*  (AND*  (ADJ  LW ALL  PIC) 
(PICTURE  PIC) 

((OPTION  (DOOR  L*  ALL ) (WALL  LwALL))) 

( (AND*  (ADJ  LW ALL  PIC)  (DOOR  LWALL))) 

((OP*  (DOOR  L’WALL)  (DOOR  RoALL))) 


VARIABLE:  ( WALL  LWALL) 

VALUE:  .626 

proce  di'PE  : 


RELATIVES: 


(AND*  .773  (VOT*  (OR*  (WALL  DR) 

(WALL.  PIC) 
(WALL  CRACK) 
(WALL  WbsKT] 

((OPTION  (DOOR  LWALL)  (WAIL  LWALI  ) ) ) 
((OR*  (WALL  LWALL)  (WALL  RWALL))) 


VARIABLE;  (DOUR  CBACK) 


PROCEDURE : 


RELATIVES: 


(DOOR  CBACK) 

((OPTION  (DOOR  CBACK)  (CHAIRBACK  CBACK)  (WALL  CRACK))) 
((OR*  (DOOR  DP)  (DOOR  PIC)  (DOUR  CBACK)  (DOOR  WBSKT ) ) ) 


VAPTABLE:  (CHAIRBACK  CBACK) 

VALUE;  .11 

PROCEDURE : 

(CHAIRBACK  CRACK) 

RELATIVES: 

((OPTION  (DOOR  CBACK)  (CHAIRBACK  CBACK)  (WALL  CBACK))) 


VARIABLE:  (WALL  CBACK) 

VALUE:  0.0 

PROCEDURE: 

(WALL  CBACK) 

1 RELATIVES: 

FIGURE  8 DATABASE  AT  EQUILIBRIUM  PRIOR  TO  SEARCH  (Continued) 
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<TENENBAUM>BAYES.RPT;2 


WED  4-FEB-7b  1:53PR 


Pag*  2:2 


* 


00103 

((OPTION  (DOOR  CBACK)  (CHAIRBACK  CBACK ) 

(WALL  CBACK)  ) ) 

00104 

( ( OR*  (WALL  DR)  (WALL  PIC)  (WALL  CBACK) 

(WALL  WBSKT  ) I ) 

00105 

0010b 

00107 

VARIABLE: 

(PICTURE  PIC) 

00108 

VALUE: 

.251 

00109 

PROCEDURE: 

1 ' 

00110 

(AND*  .3  (NOT*  (OR*  (AND*  (ADJ  RwALL  PIC) 

00111 

(DOOR  RWALL)  ) 

00112 

(AND*  (ADJ  LwALL  PIC) 

00113 

(DOUR  LWALL) 

00114 

RELATIVES: 

00115 

((OPTION  (PICTURE  PIC)  (DOOR  PIC)  (WALL 

PIC)  ) ) 

0011b 

((AND*  (ADJ  RWALL  PIC)  (PICTURE  PIC))) 

00117 

((AND*  (ADJ  LWALL  PIC)  (PICTURE  PIC))) 

001  18 

001  19 

00120 

VARIABLE: 

(DOOR  PIC) 

00121 

VALUE: 

0.0 

0012? 

frocedupe: 

00123 

(POOR  PIC) 

00124 

RELATIVES: 

00125 

((OPTION  (PICTURE  PIC)  (DOOR  PIC)  (WALL 

PIC))) 

0012b 

( ( OR*  (POOP  DR)  (DOOR  PIC)  (DOOR  CBACK) 

(DOOP  WBSKT)  ) ) 

00127 

00128 

00129 

VARIABLE: 

(WALL  PIC) 

[ • 

001  30 

VALUE: 

U .0 

00131 

PROCEDURE: 

00132 

(WALL  PIC) 

00133 

PFLAT IVES: 

001  34 

((OPTION  (PICTURE  PIC)  (DOUR  PIC)  (WALL 

PIC)  ) ) 

00135 

((OR*  (WALL  PR)  (WALL  PIC)  (WALL  CBACK) 

(WALL  WBSKT))) 

001  3b 

001  37 

00138 

VARIABLE: 

(DOOP  DP) 

001  39 

VALUE: 

.175 

* ' * 

00140 

FROCEDUPE: 

00141 

(AND*  .227  (ANO*  (NOT*  (OR*  (DOOR  LWALL) 

00142 

(DOOR  RWALL))) 

00143 

(NOT*  0.0))) 

* 

00144 

PELATIVES: 

* h- 

00145 

((OPTION  (DOOR  DR)  (WALL  DP))) 

0014b 

((OR*  (DOOR  DP)  (DOOR  PIC)  (DOOP  CBACK) 

(DOOR  WBSKT))) 

00147 

1 ** 

00148 

- 

00149 

VAPIABLE: 

(WALL  DR) 

00150 

VALUE: 

.142 

00151 

PROCEDURE: 

*. 

00152 

(AND*  .773  (NOT*  (OR*  (WALL  LwALL) 

00153 

(WALL  RWALL) 

3 

00154 

RELATIVES : 

f ^ 

00155 

((OPTION  (DOOR  DP)  (WALL  DP))) 

ip 

0015b 

( (UR*  (WALL  DR)  (WALL  PIC1  (WALL  CRACK) 

(WALL  V BSKT ) ) ) 

N 

i,  , 

n 
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fc 

that  tht-  bright  region  LWALL  is  door  by  an  amount  proportional  to  the 
likelihood  that  any  of  the  dark  regions  DR,  PIC,  or  CBACK  are  thought  to 


be  door.  Incidentally,  an  identical  negated  disjuntion  supports  the  con- 
straint ( HOMOGENEOUS  DOOR)  in  the  likelihood  function  of  the  region 
interpretation  (DOOR  RWALL),  since  regions  RWALL  and  LWALL  have  similar 
brightnesses . 

The  third  term  (NOT*  (AND*  (ADJ  LWALL  PIC) (PICTURE  PIC))),  sup- 
ports the  constraint  (NOT  (ADJACENT  PICTURE  DOOR)).  This  term  reduces  the 
likelihood  that  region  LWALL  has  the  interpretation  DOOR,  proportional  to 
the  likelihood  that  region  PIC  has  interpretation  PICTURE  and  the  degree 
to  which  regions  PIC  and  LWALL  are  adjacent.  The  other  likelihood  functions 
shown  in  Figure  8 are  composed  of  similar  terms  supporting  applicable 
homogeneity  or  adjacency  constraints. 

2 . Numerical  Evaluation  of  Likelihoods 

The  numerical  evaluation  of  likelihood  functions  requires  that 
conventions  for  combining  evidence  be  established.  Two  simple  schemes 
were  considered  . The  first  scheme  is  a set  theoretic  formulation  that 
treats  interpretation  likelihoods  and  region  relation  values  as  independent 
probabilities.  A conjunction  of  likelihoods  evaluates  to  the  product  of 
those  likelihoods,  the  negation  of  a likelihood  to  one  minus  the  likelihood, 
and  a d is j unction  of  likelihoods  to  one  minus  a product  of  the  negations  of 
the  1 ikelihoods . 


The  second  scheme  treats  likelihood  functions  as  definitions 
of  fuzzy  sets.  Following  Zadeh's  conventions*5,  a conjunction  of  likeli- 
hoods evaluates  to  the  minimum  likelihood,  a disjunction  of  likelihoods 
evaluates  to  the  maximum  likelihood,  and  a negated  likelihood  evaluates 
to  one  minus  the  likelihood. 

A few  qualitative  remarks  can  be  made  contrasting  the  evaluation 
of  likelihood  functions  using  the  set  theoretic  and  fuzzy  set  formulations. 
Since  likelihood  functions  are  composed  of  conjunctions  of  constraint  terms, 
the  likelihood  of  an  interpretation  is  limited,  in  both  formulations,  by  its 
least  satisfied  constraint  . In  particular,  an  interpretation  can  effectively 


<*■>  be  ruled  out  (i.e.,  its  likelihood  forced  to  zero)  by  a single  badly 


violated  constraint.  The  set  theoretic  approach  penalizes  the  likelihood 


of  an  interpretation  that  is  supported  by  a large  number  of  moderately 
satisfied  constraints,  while  the  fuzzy  set  approach  does  not.  Each  con- 
straint term  in  a likelihood  function  is  typically  represented  by  a 
disjunction  of  terms  expressing  alternative  ways  of  satisfying  the  constraint. 
The  strength  of  a constraint  in  the  fuzzy  set  formulation  will  thus  equal 
the  support  provided  by  the  individual  interpretation  that  best  satisfies 
the  constraint.  The  set  theoretic  approach,  on  the  other  hand,  yields  a 
strength  for  constraints  that  is  strictly  greater  than  the  support  provided 
by  any  individual  interpretation.  Tnis  characteristic  leads  to  overesti- 
mates of  terminal  likelihoods,  which  is  consistent  with  requirements  for 
optimal  search  stated  earlier. 

A number  of  more  elaborate  ways  for  summing  evidence  have  appeared 
recently  in  the  literatures  of  scene  analysis,  speech  understanding,  and 
diagnosis7-12 . Yakimovsky 's7  Bayesian  formulation  of  interpretation 
probabilities  is  theoretically  pleasing  but  suffers  from  the  difficulty 
of  obtaining  realistic  conditional  probabilities  in  complex  scene  domains. 
Barrow  and  Popplestone8  describe  an  ad  hoc  method  of  evaluating  conjunctions 
of  constraints  that  is  based  on  the  number  of  constraints  that  are  violated 
and  the  seriousness  of  the  violation.  Their  evaluation  gives  preference 
to  interpretations  with  many  partially  satisfied  constraints  over  inter- 
pretations with  fewer  constraints  that  are  more  completely  satisfied. 

Shortliffe0  described  a quantification  of  inexact  reasoning  in 
medical  diagnosis  in  terms  of  confirmation  theory.  His  formulation  has 
many  desirable  features  as  a basis  for  evaluating  competing  hypotheses  in 
scene  interpretation.  Individual  constraints  make  independent  contributions 
to  belief  in  a hypothesis  and  therefore  can  be  acquired  incrementally. 
Moreover,  since  evidence  for  and  against  a hypothesis  is  treated  independently, 
hypotheses  are  not  penalized  for  missing  features  (which  may  be  optional  or 
occluded)  or  for  a large  number  of  partially  satisfied  constraints  in  the 
absence  of  specific  contradictory  evidence.  Duda12  has  recently  devised  a 
new  Bayesian  approach  to  the  combination  of  evidence  that  appears  to  eliminate 
some  potential  discontinuities  in  Shortliffe's  likelihood  computations.  A 
more  sophisticated  approach  may  be  adopted  if  future  experiments  prove  our 
simple  set  theoretic  approach  inadequate.  So  far,  the  choice  of  evaluation 
does  not  appear  to  be  critical. 


25 


Compilation  Process 


r* 


3 . 


Every  region  interpretation  in  the  image  is  represented  by  its 
own  likelihood  function.  The  process  of  compiling  a likelihood  function 
for  a region  interpretation  begins  by  retrieving  the  applicable  generic 
constraints.  A support  term  is  formuluted  for  each  applicable  constraint, 
expressing  how  well  that  constraint  is  satisfied  by  the  other  region 
interpretations  in  the  image.  These  support  terms  are  then  combined  with 
the  a priori  likelihood  in  a function  that  expresses  an  upper  bound  on  the 
overall  likelihood  of  the  interpretation. 


Formulation  of  the  support  term  for  a constraint  involves  searching 

the  image  to  determine  all  region  interpretations  that  could  potentially 

satisfy  the  constraint  in  a terminal  state.  Details  of  the  search  vary, 

however,  depending  on  the  type  of  constraint.  For  standard  relational 

constraints,  all  regions  admitting  the  required  interpretations  are  tested 

to  determine  whether  they  also  obey  the  required  region  relation  (e.g., 

above  and  adjacent)  with  respect  to  the  constrained  region.  Region  inter- 

* 

pretations  that  pass  are  represented  in  the  support  term  for  the  constraint 
by  a conjunction  of  the  interpretation  likelihood  and  the  strength  with 
which  the  region  relations  was  satisfied. 


Support  terms  for  constraints  that  do  not  fit  the  standard 
relational  format  are  compiled  by  special  procedures  that  are  provided  for 
each  such  constraint.  These  support  gathering  procedures  are  called  with 
the  constrained  region  interpretation  as  an  argument  and  return  a support 
term  that  is  inserted  directly  into  the  top-level  conjunction  of  the  like- 
lihood function.  Such  a procedure  was  invoked,  for  example,  in  compiling 
the  support  term  for  the  constraint  (HOMOGENEOUS  DOOR)  in  the  likelihood 
function  of  the  region  interpretation  (DOOR  LWALL) . This  procedure  first 
retrieves  all  region  interpretations  in  the  current  image  containing  the 
constrained  interpretation,  in  this  example  (DOOR  LWALL),  (DOOR  DR), 


* 

As  an  expedient,  a region  interpretation  is  only  considered  as  potential 
support  for  a constraint  if  the  strength  of  the  corresponding  region 
relation,  which  is  static  exceeds  0.1. 


(DOOR  PIC),  and  (DOOR  RWALL) . The  procedure  then  forms  a negated  disjunction 
containing  the  subset  of  these  whose  regions  differ  in  brightness  from  that 
of  the  constrained  region  (LWALL)  by  more  than  10%. 

The  support  term  for  the  constraint  (HOMOGENEOUS  WALL)  in  the 
likelihood  function  of  (WALL  LWALL),  line  72  in  Figure  8,  is  compiled  using 
the  same  procedure,  which  now  operates  by  retrieving  all  region  interpretations 
containing  the  constrained  interpretation  WALL. 

Compound  constraints,  such  as  (NOT*  (ADJACENT  PICTURE  WALL))  and 
(OR*  (ADJACENT  PICTURE  WALL) (ADJACENT  PICTURE  FRAME))  are  first  parsed  into 
their  elementary  constituent  constraints.  Support  terms  are  obtained  inde- 
pendently for  each  constituent.  These  support  terms  are  then  inserted  back 
into  the  original  compound  constraint  in  place  of  the  corresponding  constituent. 
The  resulting  compound  term  expresses  the  support  that  exists  for  the  ori- 
ginal compound  constraint.  For  example,  consider  the  constraint  (NOT* 

(ADJACENT  PICTURE  DOOR))  applied  to  the  region  interpretation  (DOOR  RWALL) 
in  Figure  4.  The  term  (AND*  (ADJACENT  PIC  RWALL) (PICTURE  PIC))  was  first 
formulated  to  represent  support  for  the  constituent  (ADJACENT  PICTURE  DOOR). 

(In  this  example,  region  interpretation  (PICTURE  PIC)  was  the  sole  source 
of  support.)  The  support  verm  then  replaced  the  constituent  in  the  original 
compound  constraint,  forming  the  compound  expression  (NOT*  (AND*  (ADJACENT 
PIC  RWALL) (PICTURE  PIC))).  This  expression  appears  in  the  likelihood  function 
of  region  interpretation  (DOOR  RWALL)  at  line  34  of  Figure  8. 

If  any  constraint  has  no  support  among  the  other  region  inter- 
pretations, the  likelihood  of  the  constrained  interpretation  is  pinned 
at  zero.  Alternatively,  a compilation  procedure  could  search  the  image  for  a 
new  region  with  the  needed  interpretation,  using  techniques  for  goal  directed 
search,  such  as  those  developed  by  Garvey13. 

C.  Relaxation — The  XDEMON  System 

The  likelihood  functions  must  be  evaluated  simultaneously  to  determine 
consistent  likelihood  estimates  for  all  interpretations.  Previously,  it  was 
remarked  that  evaluation  could  proceed  in  a highly  parallel  manner,  with 
each  likelihood  function  represented  by  an  independent  process.  This  parallel 
approach  has  been  efficiently  simulated  on  a serial  computer  using  asynchronous 
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parallel  processes  that  interact  through  a global  data  base.  A set  of 
LISP  function,  known  collectively  as  the  system  XDEMON , have  been  developed 
to  facilitate  creation  of  this  data  base  for  particular  constraint  problems. 
These  functions  are  documented  in  Appendix  D. 

The  global  data  base  consists  of  variables,  each  of  which  has  a value 
and  an  associated  process  that  computes  the  value  in  terms  of  the  current 
values  of  other  variables.  Each  variable  also  has  a list  of  related  vari- 
ables that  use  the  present  variable  as  input.  When  the  cumulative  change 
in  the  value  of  a variable  exceeds  a threshold,  its  related  variables  are 
reevaluated  by  adding  their  processes  to  a set  of  jobs  to  be  run.  Running 
a process  can  change  the  value  of  a related  variable,  causing  additional 
processes  to  be  activated.  Execution  terminates  when  the  job  set  is  empty. 

For  scene  interpretation,  each  possible  region  interpretation  is 
represented  in  the  data  base  (1)  by  a variable  of  the  form  (CHAIRBACK  R4) 
whose  value  is  the  current  likelihood  of  that  interpretation,  and  (2)  by 
an  associated  process  for  computing  that  likelihood  value  based  on  the  cur- 
rent likelihood  values  of  other  region  interpretations.  Likelihood  evaluation 
is  initiated  by  loading  the  job  set  with  the  processes  of  interpretations 
for  which  updated  likelihoods  are  required.  To  obtain  an  initial  set  of 
consistent  likelihoods  the  job  set  is  loaded  with  the  processes  of  every 
region  interpretation.  To  pursue  the  consequences  of  a particular  instan- 
tiation on  likelihoods  previously  in  equilibrium,  the  job  set  is  loaded 
with  the  processes  of  the  variables  on  the  related  variable  list  of  the 
instantiated  variable. 

For  efficiency,  the  processes  that  compute  likelihoods  are  decomposed 
hierarchically  into  elementary  s-expressions,  each  canonically  represented 
by  an  XDEMON  variable.  Figure  9 illustrates  this  decomposition  for  the 
likelihood  function  described  in  Figure  7.  Superexpression  variables  are 
placed  on  the  related  variable  lists  of  variables  representing  subexpressions . 
At  the  lowest  level,  subexpression  variables  representing  region  relations 
(e.g.,  (ABOVE  R1  R2>)  and  region  interpretations  (e.g.,  (CHAIRSEAT  R1 ) ) 
become  relatives  of  atomic  variables  representing  regions  (e.g.,  R3)  and 
interpretations  (e.g.,  CHAIRSEAT).  The  value  of  an  atomic  region  variable 
is  the  list  of  boundary  coordinates  of  that  region.  If  this  value  were 
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altered  (e.g.,  by  merging  or  splitting  the  region),  then  all  region  rela- 
tions and  region  interpretation  variables  in  which  the  altered  region 
variable  appeared  would  be  reevaluated. 

The  above  decomposition  increases  efficiency  by  terminating  reevaluation 
at  the  lowest  level  subexpression  whose  value  is  unchanged  by  a triggering 
event  . Suppose,  for  example,  that  the  current  strength  of  the  region 
relation  (ABOVE  R3  R2)  in  Figure  9 was  0.4  and  that  the  likelihood  of  region 
interpretation  (CHAIRLEG  R2)  was  0.5.  Assuming  fuzzy  logic,  the  superex- 
pression (AND*  (ABOVE  R3  R2) (CHAIRLEG  R2)>  in  the  lower  left  corner  of 
Figure  9 would  then  evaluate  to  0.4  (i.e.,  the  minimum  of  0.4  and  0.5). 

A jump  in  the  likelihood  of  (CHAIRLEG  R2)  from  0.5  t<>  0.6  would  trigger 
reevaluation  of  the  above  superexpression.  However,  its  value  would  be  unchanged 
and  the  reevaluation  process  would  immediately  terminate.  The  canonical  representa- 
tion of  subexpressions  in  the  decomposition  further  minimizes  redundant 
computation  in  cases  where  a subexpression  is  common  to  several  superexpres- 
sions. In  Figure  8,  for  example,  the  same  support  term  for  the  constraint 
(HOMOGENEOUS  WALL)  appears  at  lines  45  and  72  in  the  likelihood  functions 
of  the  interpretations  (WALL  LWALL)  and  (WALL  RWALL) , respectively.  The 
same  XDEMON  variable  represents  this  term  in  the  decomposed  likelihood 
functions  of  both  interpretations.  This  variable  is  reevaluated  only  once 
when  the  likelihood  of  a supporting  region  interpretation  e.g.,  (WALL  DR) 
changes . 

The  alternative  interpretations  of  a given  region  are  associated  by 
canonic  variables  of  the  form  (OPTION  (CHAIRBACK  R4) (WALL  R4)),  which 
evaluates  to  the  current  number  of  interpretations  of  the  region  whose 
likelihood  exceeds  0.1.  Option  variables  are  useful  in  situations  where 
the  competing  interpretations  must  be  manipulated  as  a set,  such  as  con- 
straints requiring  that  two  regions  take  the  same  (unspecified)  interpretation 
or  normalizations  requiring  that  the  likelihoods  of  alternative  interpretations 
sum  to  1.0.  Since  option  variables  are  reevaluated  when  any  interpretation 
likelihood  of  their  associated  region  is  altered,  many  important  administrative 
details  of  a search  can  be  handled  efficiently  as  side  effects  of  the  process 
(see  Section  III.D).  The  AND/0R/0PTI0N  structure  of  the  resulting  global 
data  base  (Figure  8)  is  remarkably  similar  to  that  adopted  independently 
for  the  HEARSAY- 1 1 Speech  Understanding  System. 14 
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Occasionally,  the  relaxation  process  will  terminate  with  one  highly 
likely  interpretation  for  each  region.  More  often,  the  interpretation  of 
one  or  more  regions  remains  ambiguous . A search  is  then  needed  to  deter- 
mine which  of  the  possible  interpretations  maximize  the  combined  likelihood 
of  the  whole  scene . 

The  search  algorithm  outlined  in  Figure  3 is  implemented  using  a 

general  state-saving  mechanism  that  allows  a current  computational  context 

* 

to  be  reinstated  at  a future  time.  The  search  context  for  M consists  of 
the  complete  network  of  variables  described  in  Section  1 1 1-C  plus  additional 
"state”  variables  that  characterize  each  search  state.  These  include  a 
score,  a list  of  previous  region  instantiations,  and  a priority  queue  of 
instantiations  yet  to  be  tried  (IQUEUE).  Search  states  in  various  stages 
of  instantiation  are  inserted  onto  a priority  queue  of  states  (SQUEUE), 
ordered  by  score.  A search  proceeds  by  reinstating  the  highest  scoring 
state  on  SQUEUE,  selecting  the  best  instantiation  from  the  current  IQUEUE, 
then  reevaluating  the  network  of  variables  in  the  new  context  created  by 
that  instantiation.  An  acceptable  solution  terminates  the  search.  Other- 
wise, IQUEUE  is  updated  and  the  resulting  state  is  added  to  SQUEUE.  The 
search  then  continues  in  the  current  highest  scoring  state. 

IQUEUE  contains  the  OPTION  variables  for  all  ambiguously  interpreted 

regions  remaining  in  a state,  ordered  by  the  likelihood  of  their  most 

probable  interpretation  (highest  first).  An  interpretation  is  hypothesized 

(Step  2 in  Figure  3)  by  popping  the  top  OPTION  variable  from  IQUEUE  and 

instantiating  the  corresponding  region  to  its  most  probable  interpretation. 

* 

This  best  first  instantiation  policy  was  borrowed  from  conventional  A 

type  search  algorithms  and  proved  adequate  for  our  simple  experiments. 

However,  the  choice  of  instantiation  in  a constraint  satisfaction  algorithm 
* 

such  as  M should  also  take  into  account  the  likelihood  that  the  instan- 
tiation will  lead  to  a quick  contradiction,  thereby  allowing  abandonment 
of  that  branch  of  the  search  tree.  The  likelihood  of  achieving  a contra- 
diction depends  on  factors,  such  as  the  number  of  alternative  interpretations 
of  the  region,  the  number  of  interpretations  directly  constrained  by  the 
instantiated  interpretation,  the  ambiguity  of  the  regions  associated  with 
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those  directly  constrained  interpretations,  and  the  number  of  interpretations 
they  in  turn  constrain.  Clearly,  finding  the  optimal  instantiation  is 
another  major  search  problem,  termed  by  Montanari  "the  secondary  optimization" 
problem . 1 " 
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Two  search  states  are  established  in  Step  3 to  explore  the  consequences 
of  both  asserting  and  denying  the  interpretation  hypothesis  selected  in 
Step  2.  A region  interpretation  is  denied  by  setting  its  local  and  current 

* 

likelihoods  to  0.0  and  removing  it  from  the  relative  lists  of  other  variables. 

An  interpretation  is  asserted  by  denying  the  alternative  interpretations 
for  that  region.  The  reason  for  hypothesizing  a single  interpretation, 
rather  than,  for  example,  splitting  the  set  of  possible  interpretations 
of  a region  into  approximately  equal  subsets,  was  to  maximize  the  likelihood 
of  forcing  a contradiction. 

The  evaluation  of  each  branch  in  the  search  (Steps  4 through  6)  is 
handled  as  in  the  uninstantiated  top  level  state,  by  executing  a job  list. 

The  job  list  initially  contains  the  set  of  variables  directly  related  to 
variables  whose  likelihoods  were  altered  in  instantiating  that  branch.  This 
initial  set  includes  the  OPTION  variable  of  the  instantiated  region.  Addi- 
tional jobs  and  option  variables  are  added  dynamically  as  a consequence  of 
reevaluating  these  initial  variables.  Processing  terminates  with  an  updated  set  of 
equilibrium  likelihoods  and  updated  values  of  the  state  variables  SCORE  and  IQUEUE . 

The  termination  tests  in  Step  5 and  the  updating  of  variables  SCORE  and 
IQUEUE  are  both  accomplished  as  side  effects  of  reevaluating  OPTION  variables. 
Specifically,  the  score  is  incremented  by  subtracting  the  previous  best 

* 

Setting  local  likelihood  to  zero  permanently  pins  an  interpretation's 
overall  likelihood  to  zero.  Removing  that  variable  from  relative  lists 
thus  avoids  unnecessary  reevaluations.  Any  variables  thereby  left  with 
no  relatives  are  themselves  removed  from  relative  lists  of  other  variables 
so  that  variable  reevaluations  will  occur  only  when  a new  value  of  the 
variable  might  be  utilized.  Note  that  the  relative  lists  of  region  inter- 
pretation variables  always  include  an  OPTION  variable  and  thus,  such  variables 
are  not  disconnected  except  when  directly  pinned  to  zero. 

•f 

In  early  experiments,  the  current  likelihood  of  the  asserted  interpreta- 
tion was  boosted  to  1.0  to  excite  the  equilibrium  process.  This  artificial 
stimulus  proved  unnecessary  and  was  abandoned  in  later  experiments. 
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interpretation  likelihood  of  the  region  (stored  on  the  property  list  of  the 
OPTION  variable)  and  adding  the  current  best  likelihood.  If  more  than  one 
interpretation  of  the  region  has  a likelihood  exceeding  0.1,  a pointer 
to  the  OPTION  variable  is  inserted  onto  IQUEUE  in  a position  determined 
by  the  current  likelihood  of  its  best  interpretation.  If  the  likelihoods 
of  all  possible  interpretations  of  a region  drop  below  0.1  the  search 
state  is  assumed  to  be  inconsistent  and  is  abandoned.  A solution  (i.e., 
a consistent  set  of  unique  interpretations)  is  indicated  if  after  evaluating 
all  OPTION  variables,  no  contradictions  have  been  detected  and  the  IQUEUE 
of  that  state  is  empty. 

Updating  state  variables  is  an  efficient  process,  because  OPTION 
variables  are  added  to  the  job  list  only  when  the  likelihood  of  an  asso- 
ciated region  interpretation  has  actually  been  altered . Moreover,  OPTION 
reevaluations,  unlike  other  jobs,  are  added  to  the  end  of  the  job  list  so 
that  evaluation  occurs  only  once  in  each  search  state  using  the  final 
equilibrium  likelihoods. 

The  nature  of  the  above  search  is  determined  by  the  functions  used 
to  update  SQUEUE  and  IQUEUE  and  by  the  termination  condition.  Alternative 
search  strategies  are  easily  instituted.  A depth  first  search,  for  example, 
is  obtained  by  always  adding  new  search  states  to  the  front  of  SQUEUE, 
while  a breadth  first  search  is  realized  by  always  adding  them  to  the  end.1' 
Heuristic  guidance  can  be  introduced  into  the  search  by  the  function  that 
updates  IQUEUE.  The  termination  condition  can  be  chosen  to  select  the 
highest  scoring  solution  (i.e.,  completely  instantiated  terminal  state), 
the  first  solution  obtained  (which,  with  arbritrary  constraints,  is  not 
guaranteed  optimal)  or  a complete  enumeration  of  all  consistent  solutions. 

E.  Using  MSYS 

All  experiments  with  MSYS  have,  to  date,  been  performed  on  manually 
partitioned  scenes.  An  interpretation  problem  is  posed  in  the  following 
way.  First,  the  experimenter,  using  a trackball,  circles,  and  names  a set 
of  test  regions  on  the  displayed  image  of  a scene  (see,  for  example, 

Figure  4) . Next  he  enters  the  constraints  to  be  used  in  the  current  exper- 
iment . He  may  also  directly  assert  symbolic  relationships  among  regions 
(e.g.,  that  two  regions  be  considered  adjacent).  This  ability  was  useful 
for  simulating  unimplemented  relational  procedures.  Interpretation  is 
initiated  by  calling  the  function  INTERPRET  with  a region  file  or  a list 
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of  regions  as  an  argument . MSYS  responds  with  a complete  protocol  of  the 
interpretation  process  containing,  first,  a list  of  locally  possible  region 
interpretations  and  their  initial  a priori  likelihoods,  second,  a trace  of 
all  jobs  executed  from  the  job  list,  and  third,  a final  list  of  unique 
region  interpretations  with  associated  likelihoods  (or  else  a message 
announcing  failure  to  find  a consistent  set  of  interpretations.)  Appendix  B 
contains  the  complete  output  protocol  for  the  room  scene  analysis  discussed 
in  Section  1 1 .C . 

Constraints  are  entered  in  the  format:  (ADDCONST  REL  VLIST) . REL  is 

a simple  relation  such  as  (ABOVE  CHAIRBACK  CHAIRSEAT),  a Boolean  expression 
of  simple  relations,  such  as  (OR  (ADJACENT  PICTURE  WALL) (ADJACENT  PICTURE 
FRAME)),  or  a functional  constraint  such  as  (HOMOGENEOUS  DOOR).  VLIST  is 
a list  of  the  interpretations  to  which  the  constraint  applies.  Thus,  (ADDCONST 
(ADJACENT  PICTURE  WALL) (PICTURE) ) requires  all  pictures  to  be  adjacent  to 
WALLS  but  puts  no  constraint  on  WALLS.  If  VLIST  is  omitted,  MSYS  assumes 
that  the  constraint  applies  mutually  to  all  interpretations  mentioned 
within  it.  Constraints  on  the  same  interpretation  specified  in  different 
ADDCONST  statements  are  embedded  in  an  implicit  conjunction.  Figure  10 
illustrates  the  actual  format  used  for  specifying  the  constraints  in  the 
example  of  Section  II. C. 

Symbolic  region  relations  are  asserted  using  the  XDEMON  SETVAL 
function  (see  Appendix  D),  which  creates  a corresponding  data  base  variable 
and  sets  its  likelihood  to  a desired  value.  For  example,  the  fact  that 
region  R1  is  adjacent  to  region  R2  could  be  asserted  with  certainty  by 
executing  (SETVAL  (ADJACENT  R1  R2)  1.0). 

F . Summary  of  Experimental  Results  in  Room  Scene  Domain 

The  results  shown  in  Section  II. C were  obtained  with  set  theoretic 
logic.  Identical  final  interpretations  were  obtained  using  fuzzy  logic.  With 
fuzzy  logic,  the  relaxation  process  converged  much  more  rapidly  in  every  state 
because  the  values  of  disjunctions  (conjunctions)  could  change  on'  • when 
the  value  of  their  strongest  (weakest)  supporting  interpretation  was  altered. 
This  advantage  was  offset  by  a reduction  in  quality  of  the  resulting 
likelihood  estimates,  and  necessitated  backtracking  during  search.  Ir. 
particular,  (WALL  DR),  an  incorrect  interpretation,  had  the  highest  overall 
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I ADDCONST  (QUOTE  (NOT-  (ADJ  DOOR  PICTURE)))) 
(ADDCONST  (QUOTE  (FUNCTION  ROOMPARTI) 

(QUOTE  (DOOR  WALLII) 

(ADDCONST  (QUOTE  (FUNCTION  HOMO)) 

(QUOTE  DOOR)) 

(ADDCONST  (QUOTE  (FUNCTION  HOMO)) 

(QUOTE  WALL)) 

(ADDCONST  (QUOTE  (FUNCTION  HOMO)) 

(QUOTE  CBACK)) 

FIGURE  10  SPECIFICATION  OF  RELATIONAL  CONSTRAINTS 
GIVEN  IN  FIGURE  2 
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likelihood  in  the  initial  equilibrium  state,  and  was,  therefore,  chosen 
as  the  first  candidate  for  instantiation.  Fortunately,  the  search  was 
side-tracked  only  briefly  because  a higher  global  score  was  obtained  in 
the  competing  context  where  (WALL  DR)  was  denied.  Thereafter,  the  search 
went  directly  to  the  solution  obtained  in  Figure  6,  and  with  significantly 
less  propagation.  A related  experiment  was  performed  using  set  theoretic 
logic  with  normalized  likelihoods  that  summed  to  1.0  over  all  possible 
interpretations  of  a region.  This  normalization  affected  neither  the 
order  of  instantiation  nor  the  final  solution.  However,  it  introduced 
many  additional  oscillations  into  the  relaxation  process,  since  the  like- 
lihoods of  all  interpretations  of  a region  had  to  be  readjusted  whenever 
any  was  reevaluated  . Normalization  was  thus  rejected  as  unnecessary  and  in- 
efficient. The  overall  conclusion  was  that  any  reasonable  rules  for 
combining  evidence  would  probably  suffice. 


IV  DISCUSSION 


In  this  section,  we  examine  our  work  in  the  general  context  of 
knowledge-based  search  and  then  suggest  some  applications. 

* 

A.  M as  a Heuristic  Search  Algorithm 
+ 

M is  basically  a conventional  heuristic  search  algorithm  that  employs 
a novel  evaluation  function  to  guide  search.  A relaxation  method  is 
invoked  after  each  instantiation  to  obtain  consistent  likelihood  estimates 
for  the  remaining  interpretation  possibilities  of  each  region.  The 
evaluation  function  forms  an  estimate  of  the  highest  scoring  terminal 
state  reachable  from  the  current  (partially  instantiated)  state,  by  summing 
the  likelihoods  of  the  highest  scoring  interpretation  possibility  of  each 
region.  The  search  then  proceeds  by  returning  to  the  state  with  the 
highest  evaluation,  and  instantiating  next,  the  highest  likelihood  inter- 
pretation of  an  uninstantiated  region. 


It  is  well  known  that  the  effectiveness  of  a genera te-and-test  search 
is  improved,  sometimes  substantially,  when  problem  constraints  are  used  for 
guiding  generation  as  well  as  for  testing.  With  deterministic  constraints, 
it  may  be  possible  to  eliminate  certain  variable  assignments  from  con- 
sideration and  thereby  reduce  the  branching  of  the  search.  For  example,  if 
it  can  be  shown  that  a variable  assignment  is  inconsistent  with  all  pos- 
sible assignments  of  another  variable,  then  all  cases  involving  that 
assignment  need  not  be  generated.  In  highly  constrained  problems,  elim- 
inations may  propagate  to  reduce  drastically  the  set  of  feasible  solutions. 
Waltz's  filtering  algorithm  provides  a well  known  and  dramatic  illustration 
of  this  phenomenon. 


With  probabliscic  constraints,  variable  assignments  cannot  be  eliminated 

absolutely.  However,  they  can  be  ordered  preferentially  so  that  a search  will 

find  the  best  solutions  first,  without  having  to  enumerate  cases  involving 

* 

unlikely  assignments.  The  relaxation  process  in  M can  be  viewed  as  the 
analog  for  probablistic  constraints,  of  Waltz's  filtering  algorithm  for 
deterministic  constraints,  where  instead  of  eliminating  inconsistent 
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assignments,  their  likelihoods  are  depressed.  The  advantage  in  both  cases, 
derives  from  the  fact  that  inconsistencies  are  detected  once,  before 
instantiation,  rather  than  having  to  be  discovered  repeatedly  on  multiple 
branches  of  the  search  tree. 

♦ 

1.  The  Admissibility  and  Optimality  of  M 

With  one  reasonable  restriction,  it  can  be  formally  proved  that 

* 

M is  admissible  (i.e.,  that  it  will  achieve  the  highest  scoring  solution 
for  a given  set  of  constraints.)  This  restriction  requires  that  the  likeli- 
hoods of  interpretations  be  based  solely  on  supportive  constraints, 
e.g.,  (ADJACENT  PICTURE  WALL),  as  opposed  to  contraindicating  constraints, 
e g.,  (NOT  (ADJACENT  PICTURE  DOOR)).  This  restriction  guarantees  that 
interpretation  likelihoods,  computed  as  per  Section  III.B,  will  be  over- 
estimates of  their  respective  likelihoods  in  any  terminal  state.  Hence, 
the  evaluation  function,  which  is  formed  by  summing  the  overestimated 
likelihoods,  will  be  an  upper  bound  on  the  score  of  the  best  terminal  state 
reachable  from  a given  partially  instantiated  state.  Moreover,  the  com- 
puted upper  bound  cannot  be  increased  by  additional  instantiations.  Under 

* 4c 

these  conditions,  M will  perform  the  eouivalent  of  an  A search  and  can 

be  formally  proved  to  be  admissible.  Appendix  A sketches  the  proof  which 

* 

is  based  on  this  analogy  with  the  admissible  A algorithm.  Continuing  the 
♦ * 

A analogy,  M is  also  optimal  in  the  sense  that  no  other  admissible 

algorithm,  using  an  evaluation  function  that  is  a weaker  upper  bound,  can 

reach  the  optimal  solution  through  fewer  partially  instantiated  states.  We 

* 

return  to  this  point  in  Section  IV. A. 3,  where  M is  compared  with  other 
search  algorithms. 

The  restriction  to  constraints  that  are  supportive  rather  than 
contradictory  is  not  a serious  one.  A cont ra i nd ica t ing  constraint  can,  in 
principle,  always  be  expressed  by  a set  of  supportive  constraints,  that 
explicitly  enumerate  the  allowed  possibilities.  For  example,  the  constraint 
(NOT  (ADJACENT  PICTURE  DOOR))  is  eouivalent  to  a set  of  constraints  of  the 
form  (ADJACENT  PICTURE  WALL),  (ADJACENT  PICTURE  FRAME),  and  (ADJACENT 
PICTURE  PICTURE)  enumerating  every  region  interpretation  that  can  be  legally 

-f 

Whether  or  not  the  highest  scoring  solution  corresponds  to  the  correct 
interpretation  of  the  scene  depends,  of  course,  on  the  sufficiency  of 
the  constraints. 
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adjacent  to  PICTURE.  If  this  set  is  too  large  to  enumerate,  a practical 
alternative,  still  satisfying  the  requirements  of  admissibility,  would  be 
a procedural  constraint  on  the  interpretation  PICTURE,  satisfied  by  the 
highest  scoring  interpretation,  other  than  DOOR  of  each  adjacent  region.  We 
therefore  conclude  that  the  restriction  to  supportive  constraints  is,  at 
worst,  a practical  limitation  affecting  the  parsimony  of  constraint 
express  ion . 

The  efficiency  of  a search  (measured  by  number  of  instantia- 
tions) can  sometimes  actually  be  improved  by  sacrificing  formal  admis- 
sibility. Experience  with  strong  contraindicat i ve  constraints  has  shown 
that  erroneous  instantiations  usually  produce  drastic  reductions  in  score 

and  consequent  early  abandonment  of  false  search  path:..  Thus,  even  with 

♦ 

nonsupport ive  constraints,  M is  still  an  effective  heuristic  search 

♦ 

algorithm.  If  a truly  optimal  solution  is  essential,  the  M algorithm 
can  be  modified  so  that  solutions  are  enumerated  exhaustively,  in  order  of 
merit,  subject  to  prior  pruning  by  a branch  and  bound  test. 

2.  Stability  and  Convergence  of  the  Relaxation  Process 
♦ 

The  M algorithm  presumes  that  the  relaxation  process  will  con- 
verge to  a stable  set  of  equilibrium  likelihood  values  following  each  stage 
of  instantiation.  However,  except  in  a few  special  cases,  almost  nothing 
can  yet  be  said  regarding  formal  criteria  for  guaranteeing  this  convergence 
The  constraint  restrictions  imposed  for  admissibility  obviously  preclude 
the  relaxation  process  from  oscillating  (since  likelihoods  can  only 
decrease)  but  they  do  not  preclude  monotonic  decay  toward  a set  of  likeli- 
hoods, all  of  which  are  zero.  Stability  proofs  have  been  formulated  in 

1 7 “*  1 6 

related  work  for  the  special  case  of  Boolean  supportive  constraints 
and  for  the  case  where  the  likelihood  of  an  interpretation  is  either  a fuzzy 
function  (composed  of  minimums  and  maximums)  or  a normalized  sum  of  the 
interpretation  likelihoods  of  other  regions  1 • The  substance  of  these 

proofs  rests  on  establishing  that  the  range  of  possible  likelihood  values 
for  each  interpre ta tion--ini t ia 1 ly  the  interval  (0,1) — can  never  diverge 
on  any  iteration. 

The  above  convergence  proofs  concern  constraints  that  were  applied 


r ~~ 

homogeneously  to  all  problem  variables.  Convergence  behavior  for  the 

♦ 

arbitrary,  procedural ly  represented  constraints  allowed  in  M can  for  now 
be  established  only  empirically.  In  practice,  we  are  not  concerned  with 
actual  convergence  (since  relaxation  can  always  be  arbitrarily  terminated), 
but  only  with  the  asymptotic  behavior  of  the  likelihoods  after  a reasonable 

ly 

number  of  iterations.  As  Rosenfeld  pointed  out,  the  relaxation  process 
can  be  thought  of  as  a process  for  "enhancing''  the  likelihoods  of  correct 
interpretations.  Our  experiments  involved  constraints  chosen  without 
\ • specific  regard  for  convergence  and  in  all  cases  the  relaxation  process 

had  the  desired  effect  of  enhancing  the  likelihoods  of  the  correct  inter- 
pretations. These  specific  examples  tend  to  support  the  intuition  that 
| in  highly  constrained  problems,  the  correct  interpretations  will  assert 

themselves  via  the  relaxation  process. 

3.  Relation  to  Other  Search  Algorithms 


Many  previous  scene  interpretation  algorithms  can  be  viewed  as 
* 

special  cases  of  M . Waltz's  algorithm,  for  example,  is  equivalent  to 
* 

performing  M with  deterministic  supportive  constraints,  and  a priori 

likelihoods  of  unity  for  all  interpretation  possibilities.  With  these 

restrictions  (which,  incidentally,  satisfy  the  requirements  for  an 
* 

admissible  A search),  propagation  is  limited  to  cases  where  an  inter- 
pretation's likelihood  can  be  deduced  to  be  zero.  Moreover,  since  the 
likelihoods  of  possible  interpretations  remain  equal  (namely  1.0),  they 
cannot  be  used  in  selecting  which  interpretations  to  instantiate. 

The  tree  searching  algorithms  of  Dud a"0 , Guzman'1,  and  Yakimovsky 

* 

can  also  be  viewed  as  special  cases  of  M . Each  is  equivalent  to  performing 
* 

M with  restrictions  on  the  extent  to  which  the  consequences  of  likeli- 
hood reevaluations  are  propagated.  With  supportive  constraints,  thesc- 

restrictions  on  propagation  only  raise  likelihoods  that  are  already  over- 

* 

estimates.  Therefore,  invoking  the  A analogy  developed  in  Section  V.A.l, 

these  algorithms  will  perform  searches  that  are  less  directed  (as  measured 

♦ 

by  the  number  of  partially  instantiated  states  examined)  than  will  M , 
given  comparable  constraints. 

The  algorithms  of  Duda  and  Yakimovsky  illustrate  the  drawbacks 
resulting  from  restrictions  on  the  use  of  constraints.  Duda’s  algorithm. 
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performed  an  A tree  search,  but,  unlike  M , based  its  evaluation  of 
search  states  and  its  selection  of  instantiations  entirely  on  the  uncon- 
strained a priori  likelihoods  of  interpretations.  Deterministic  constraints, 
which  specified  legally  adjacent  interpretations,  were  used  only  alter 
instantiations  to  check  whether  the  assigned  interpretation  was  consistent 
with  interpretations  previously  assigned  to  adjacent  regions. 


This  limited  use  of  constraints  frequently  necessitated  redundant 
deductions.  For  example,  a region  could  get  instantiated  to  an  interpre- 
tation that  was  inconsistent  with  all  possible  interpretations  of 
an  adjacent  region  and  that  inconsistency  would  not  be  discovered  until 
each  of  those  interpretations  had  been  individually  instantiated.  Any 

intervening  instantiations  of  a third  region  would  simply  be  wasted  work. 

* 

M , by  contrast,  would  have  avoided  instantiating  the  inconsistent  inter- 
pretation in  the  first  place . 

Yakimovsky's  algorithm  performed  an  exhaustive  depth  first 
enumeration  of  possible  region  assignments;  interpretations  were  assigned 
to  regions  in  order  of  maximum  likelihood  and  a branch  and  bound  technique 
was  used  to  prune  unpromising  branches.  When  an  interpretation  was  instan- 
tiated, its  likelihood  was  frozen  and  that  value  was  used  to  update  the 

likelihoods  of  interpretations  that  were  directly  constrained  and  not  yet 

* 

instantiated . Un like  M , Yakimovsky's  algorithm  did  not  use  uninstantiated 
interpretations  to  update  a priori  likelihoods,  did  not  update  the  likeli- 
hoods of  interpretations  after  instantiation,  and  did  not  propagate  the 
consequences  of  a likelihood  reevaluation  beyond  the  interpretations 
directly  constrained  to  an  instantiated  interpretation.  For  these  reasons, 

Yakimovsky's  likelihood  estimates  are  less  informed  (i.e.,  greater  upper- 

* 

bounds)  than  those  that  M would  derive  from  the  same  knowledge.  Moreover, 
because  the  likelihoods  of  interpretations  were  arbitrarily  frozen  at  the 
time  of  instantiation,  the  terminal  likelihoods  in  each  completely  instan- 
tiated state  represent  only  an  approximate  solution  to  the  constraints. 
Conceivably,  the  algorithm  could  thus  converge  on  a false  optimum. 

j 4 . Cost-Effectiveness 

The  number  of  instantiations  is,  of  course,  only  one  measure  of 

«•> 

\ search  effectiveness.  The  computational  effort  expended  in  the  relaxation 
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process  must  be  considered  in  assessing  whether  the  M algorithm  actually 
achieves  a cost-effective  reduction  in  search. 

* 


The  cost-effectiveness  of  M rests,  intuitively,  on  many  problem 
dependent  factors,  including  the  types  of  constraints,  the  representation 
chosen  to  express  those  constraints  in  MSYS,  and  the  connectedness  of  the 
resulting  constraint  network.  The  types  of  constraints  bear  on  the  dif- 
ficulty of  constructing  MSYS  problem  representations  and  on  the  cost- 
effectiveness  of  the  constraint  in  reducing  search.  As  an  illustration 
of  the  trade-offs,  it  is  easier  to  create  an  MSYS  representation  for  a 
homogeneity  constraint  that  applies  only  to  adjacent  regions  admitting 
the  constrained  interpretation  than  for  one  that  applies  to  every  region 
in  the  scene  admitting  that  interpretation.  However,  a local  homogeneity 
constraint  would  be  less  effective  in  detecting  global  contradictions. 


The  choice  of  representation  determines  size  of  the  search  space, 
selectivity  of  constraint  propagation,  and  cost  of  constraint  execution. 
These  criteria  are  often  contradictory.  For  example,  a representation 
based  on  sets  of  possible  interpretations  for  each  region  naturally  allows 
very  efficient  execution  of  constraints  that  operate  on  the  entire  set 
(e.g.,  constraints  that  intersect  the  possible  interpretations  of  adjacent 
regions).  On  the  other  hand,  a set  representation  hinders  the  selective 
propagation  of  constraints  on  individual  interpretations.  The  connectedness 
of  the  constraint  network  affects  how  far  the  consequences  of  a likelihood 
reevaluation  can  propagate. 

Although  the  above  factors  appear  important,  we  have  so  far  been 

* 

unable  to  formulate  crisp  criteria  for  predicting  whether  M will  prove 

cost-effective  for  a particular  problem.  The  computational  complexity  of 

such  a determination  may  be  of  the  same  order  of  difficulty  as  solving  the 

original  problem.  Our  guess  is  that  in  scene  domains  with  sparse  local 

constraints,  the  limited  lookahead  employed  by  Duda  and  Yakimovsky  will 

* 

be  more  cost-effective  than  the  global  approach  of  M . On  the  other  hand, 
shallow  lookahead  is  of  little  value  in  situations  characterized  by  a 
dense  network  of  highly  interacting  constraints.  The  line-drawing  domain 
chosen  by  Waltz  is  a perfect  example  of  this  latter  class  wherein  almost 
all  of  the  problem  reduction  was  accomplished  not  by  search  but  by 
globally  propagating  the  effects  of  local  constraints. 
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Experimental  Comparison 


5 . 

All  of  the  algorithms  described  above  can  be  emulated  in  MSYS 
by  making  minor  modifications  to  the  constraint  propagation  mechanism. 

Such  emulation  provides  a fair  basis  for  experimental  comparisons  of 
performance.  Actual  emulations  of  the  Waltz  and  Duda  algorithms  have 
been  performed  and  are  described  in  Appendices  E and  F.  These  emulations 

showed  that,  at  least  for  one  particular  set  of  constraints,  the  Waltz 

* 

and  M algorithms  were  both  more  effective  than  Duda's  algorithm.  A 
systematic  comparison  of  all  the  algorithms  on  significantly  more  complex 
scenes  is  planned  for  the  future. 

B.  MSYS  as  a System  Organization  for  Knowledge  Based  Search 

* 

MSYS,  the  implementation  of  M in  XDEMON,  has  a number  of  desirable 
attributes  as  a system  organization  for  knowledge-based  problem  solving 
and  perception.  All  knowledge  resides  in  a global  data  base  that  is 
accessible  to  all  parts  of  the  system.  Declarative  and  procedural  knowl- 
edge as  well  as  the  alternative  interpretation  hypotheses  are  represented 
uniformly  by  XDEMON  variables.  Representing  competing  hypotheses  explicitly 
in  the  data  base  where  they  are  freely  available  has  many  advantages  over 
hiding  them  in  the  internal  variables  of  a backtracking  program. 

Processes  representing  individual  elements  of  knowledge  can  be  added 
or  removed  dynamically  with  incremental  changes  in  system  performance. 
Maintenance  operations,  such  as  likelihood  updating  and  consistency  checking, 
can  be  handled  directly  by  activating  related  variables.  This  modularity 
is  essential  for  assimilating  new  knowledge  whether  from  human  experimenters 
or  an  automatic  learning  module.  It  also  allows  a system  to  construct  a 
working  data  base  by  drawing  relevant  constraints  from  a much  larger  store 
of  general  knowledge  as  the  analysis  evolves. 

Control  propagates  throughout  the  data  base  in  a highly  efficient 
data-directed  manner,  following  the  currently  most  promising  lines  of 
deduction.  Past  decisions  are  reevaluated  only  when  directly  affected  by 
subsequent  ones.  Strategy  and  demon  processes  can  be  included  on  the 
relative  lists  of  interpretation  variables  to  introduce  goal  direction. 

Goal  direction  can  also  be  imposed  by  prioritizing  the  job  set  based  on 
interest  and  expectation  associated  with  each  job. 
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XDEMON  is  a practical  realization  of  many  philosophical  objectives 
expressed  in  recent  artificial  intelligence  literature,  notably  uniformity 
of  representation,  modularity,  and  distributed  control  Similar 

system  Organizations  have  been  previously  reported  in  connection  with  work 
on  speech  understanding, 14  simulation, 2h  and  parallel  computation.11'  Our 
major  contribution  was  a demonstration  of  how  a data  driven  system  organ- 
ization could  be  used  effectively  in  performing  an  optimizing  search. 

The  asynchronous  interaction  of  knowledge  processes  in  XDEMON  suggests 
interesting  possibilities  for  parallel  implementation.  In  an  extreme 
example,  every  XDEMON  variable  could  be  represented  by  an  asynchronous 
microprocessor  that  computes  new  values  whenever  the  value  of  subordinate 

variables  change.  The  same  restrictions  that  guarantee  convergence  of 

* 

the  M relaxation  process,  namely,  that  with  supportive  constraints,  the 
range  of  likelihood  values  for  an  interpretation  cannot  diverge,  also 
guarantees  that  this  parallel  implementation  will  be  free  of  race  hazards. 
The  primary  technical  difficulty  in  such  an  implementation  would  be  the 
reconfiguration  of  interconnections  among  the  microprocessors  needed  to 
accommodate  different  constraint  problems. 


C . Applications  to  Scene  Analysis 

Scene  analysis  is  the  combined  process  of  partitioning  a scene  into 
regions  corresponding  to  meaningful  entities  and  correctly  interpreting 
those  regions.  Formally  stated,  the  objective  is  to  maximize  the  joint 
likelihood  that  region  i has  interpretation  j over  all  partitions  of  the 
scene  into  regions  and  all  assignments  of  interpretations  to  regions. 
Although  partitioning  logically  precedes  interpretation,  these  two  processes 
must  be  tightly  integrated  in  order  to  achieve  the  formal  objective. 

A recent  paper  by  Tenenbaum  and  Barrow28  presents  a way  of  using  an 
interpretation  mechanism  such  as  MSYS  to  guide  segmentation.  In  this 
approach,  knowledge  from  a variety  of  sources  is  used  to  make  inferences 
about  the  interpretations  of  regions,  and  regions  are  merged  in  accordance 
with  their  possible  interpretations.  A scene  is  first  partitioned  into 
elementary  regions  consisting  of  individual  pixels  or,  perhaps,  groups  of 
adjacent  pixels  with  identical  attributes.  Beginning  with  this  partition, 


the  system  first  performs  the  most  complete  interpretation  possible  in 

1 

«*  the  current  partition.  Based  on  this  interpretation,  it  next  merges  a 


pair  of  adjacent  regions  that  are  least  likely  to  represent  distinct  objects. 
The  process  then  iterates  by  revising  the  interpretation  to  fit  the  current 
partition  and  performing  another  merge.  As  the  partition  develops,  region 
boundaries  approach  actual  object  boundaries,  allowing  interpretation  to 
be  refined  . 

In  the  present  implementation,  the  deduction  of  region  interpretations 
is  performed  using  a limited  but  fast  version  of  MSYS,  coded  in  Fortran, 
which  allows  only  deterministic  constraints  between  adjacent  regions. 
Constraints  are  expressed  in  tabular  formats  that  specify,  for  each  possible 
region  interpretation,  the  allowed  interpretations  for  an  adjacent  region 
in  a given  relationship  (e.g.,  ABOVE,  BESIDE,  and  INSIDE).  Deduction 
proceeds  as  in  Waltz's  algorithm  by  eliminating  possible  region  interpre- 
tations that  are  not  consistent  with  any  possible  interpretation  of  an 
adjacent  region.  A second  implementation  is  planned  providing  the  full 
generality  of  MSYS,  real-valued,  procedurally  represented  constraints  among 
arbitrary  regions. 

Virtually  all  of  the  scene  interpretation  research  performed  to  date 
has  involved  narrow  semantic  domains  with  specific  expectations  (e.g., 
each  region  was  known  to  be  one  of  a dozen  or  so  possible  objects) . While 
not  arguing  against  the  use  of  expectations  when  available,  it  is  important 
to  ask  whether  the  MSYS  approach  can  be  generalized  to  work  in  real 
world  domains  containing  millions  of  objects  and  millions  of  relations. 
Clearly,  a more  concise  initial  symbolic  description  at  a general  level 
is  required,  perhaps  in  terms  of  surface  characteristics  such  as  curvature 
(planar,  convex,  and  concave),  orientation  (vertical,  horizontal),  texture 
and  material  (e.g.,  metal,  plastic,  and  wood)  and  relations  such  as  occlu- 
sion, joining,  and  support. 

The  accurate  determination  of  these  surface  characteristics  requires 
the  same  type  of  global  reasoning  used  to  deduce  object  interpretations 
but  is  based  on  general  knowledge  about  shadows,  illumination  sources, 
relative  depth,  occlusion,  surface  orientation,  texture  gradients,  and 
so  forth,  cmmon  to  many  domains.  For  example,  the  perceived  color  of  a 
region  depends  primarily  on  its  spectral  reflectance  characteristics  but 
also  upon  the  incident  illumination,  which  in  turn  is  affected  by  the 
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spectral  characteristics  and  orientations  of  nearby  surfaces.  Consequently, 
the  accurate  color  interpretation  of  one  region  may  require  the  simultaneous 
global  interpretation  of  color  and  orientation  over  all  regions.  Similarly, 
relative  depth,  in  the  absence  of  range  data,  must  be  inferred  from  the 
global  consistency  of  partial  depth  orderings  established  by  local  cues, 
such  as  T-joints  and  texture  gradients.  Once  the  scene  has  been  analyzed 
in  general  terms  that  analysis  can  be  used  to  guide  a more  detailed  spe- 
cific analysis . 

, D . Applications  to  Problem  Solving 

* 

The  M algorithm  and  its  implementation  in  XDEMON  are  useful  in  other 
areas  of  artificial  intelligence.  Scene  interpretation  typifies  a broad 
class  of  problems  in  which  values  must  be  assigned  to  variables  subject 
to  constraints  among  the  variables . This  class  encompasses  symbolic 
constraint  satisfaction  problems,  such  as  language  parsing,  line-drawing 
interpretation,  and  certain  puzzles  (e.g.,  Cryptarithmetic,  Instant  Insanity, 
Fifteen  Puzzle)  as  well  as  constraint  optimization  problems  such  as  diag- 
nosis, data  interpretation  (e.g.,  mineral  exploration)  and  many  design 
tasks  (e.g.,  architectural  layout). 

We,  and  others,  have  previously  recognized  the  utility  of  global 
propagation  techniques  in  simplifying  symbolic  constraint  satisfaction 
problems  .4  , 29-31  These  problems  often  confound  conventional  heuristic 
search  algorithms  because  of  large  search  spaces,  which  are  not  signifi- 
cantly reduced  by  individual  instantiations.  Moreover,  there  is  usually 
little  heuristic  guidance  available  for  choosing  hypotheses.  An  excellent 
example  of  a problem  with  these  characteristics  is  given  in  Appendix  G 
along  with  a sketch  of  how  MSYS  was  configured  to  solve  it. 

. Our  work  extends  the  use  of  propagation  techniques  to  constraint 

optimization  problems  opening  up  a wide  range  of  real-world  applications. 
These  problems  in  common  with  perception,  are  characterized  by  the  added 

complexities  of  noisy  data,  probablistic  constraints,  and  multiple  solu- 

..  * 

t tions  of  varying  utility.  M seems  especially  well  suited  for  cooperative 

(man-machine)  problem  solving  in  tasks  such  as  diagnosis;  the  equilibrium 

process  can  dynamically  adjust  the  relative  likelihoods  of  competing  hypo- 

K 

i, ' theses  to  reflect  new  constraints,  evidence,  and  hypotheses  that  may  be 

gl 

k interjected  by  the  man  at  any  time. 
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The  related  variables  concept  underlying  XDEMON  has  additional 
applications  in  planning  and  problem  solving  besides  constraint  propagation. 
As  one  example,  consider  the  problem  of  determining  a cost-effective 
execution  path  through  an  AND-OR  planning  graph.  The  utilities  of  subgoals 
are  interdependent  and  may  vary  with  knowledge  acquired  during  the  course 
of  execution  making  detailed  elaboration  of  alternative  execution  sequences 
unwarranted.  Garvey1  describes  an  approach,  first  implemented  in  XDEMON, 
in  which  the  utilities  of  interacting  subgoals  are  relaxed  to  equilibrium 
to  determine  the  next  best  think  to  do.  If  execution  of  the  selected 
subgoal  revises  the  utilities  of  remaining  subgoals,  the  relaxation  process 
can  b<  repeated  t"  update  priorities  before  selecting  the  subsequent  step. 
Thi-.  incremental  approach  is  particularly  appropriate  for  information 
gathering  strategies  where  incomplete  knowledge  is  inherent  at  each  step 
of  t he  plan. 

XDEMON  m also  serve  as  a simple  event  driven  simulation  language 
fot  establishing  interacting  cause  and  effect  relationships  in  an  analogue 
manner.  J.  S.  Drown" : lias  remarked  that  simulation  is  most  appropriate 

as  an  inference  technique  in  those  situations  where  conventional  ''linear" 
deductive  reasoning  breaks  down;  i.e.,  where  the  consequences  of  an  action 
lead  to  complex  side  effects  including  feedback  type  interactions  that 
can  alter  the  state  on  which  a deduction  was  initiated.  Such  a situation 
motivated  Brown's  use  of  simulation  to  deduce  the  consequences  of  faults 
introduced  into  an  electrical  network.  The  interaction  among  circuit 
variables  in  a network  resembles  the  interaction  among  interpretation 
likelihoods  in  a partitioned  scene,  which  may  account  for  resemblences 
among  the  relaxation  process  in  MSYS  and  a simulation  program. 

The  versatility  of  XDEMON  suggests  the  inclusion  of  a similar 
subroutine  package  in  artificial  intelligence  languages  such  as  QLISP.22 

E . Summary 

Scene  interpretation  typifies  a broad  class  of  problem-solving  tasks 

involving  the  assignment  of  values  to  variables  that  are  mutually  constrained 

* 

A general  constraint  optimization  algorithm,  M has  been  presented  that 
for  sufficiently  constrained  problems  is  more  powerful  than  conventional 
heuristic  search  methods,  many  of  which  can  be  treated  as  special  cases 
of  this  algorithm. 
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The  algorithm  is  based  on  the  notion  of  representing  alternative 
hypotheses  and  constraints  as  (simulated)  asynchronous  parallel  processes. 


These  processes  interact  in  a dynamic  equilibrium  that  establishes  the 
relative  likelihoods  of  competing  hypotheses.  The  equilibrium  process 
serves  as  a new  kind  of  global  lookahead  that  improves  the  order  of 
instantiation  and  context  selection  at  each  stage  of  a best  first  search. 

* 

The  admissibility  and  stability  of  the  M algorithm  have  been  for- 
mally proved  for  a restricted  class  of  constraints  as  has  its  optimality 
compared  with  conventional  ^search  algorithms,  measured  by  number  of  instan- 
tiations. Open  theoretical  ' issues  include  a precise  characterization 
of  the  class  of  "sufficiently  constrained”  problems  for  which  the  method 
is  computationally  cost-effective  and  explicit  criteria  for  selecting 
the  best  problem  representation,  constraints,  and  instantiation  order. 

The  algorithm  has  been  applied  successfully  to  both  scene  interpre- 
tation problems  and  constraint  satisfaction  puzzles.  However,  the  scene 
interpretation  experiments  should  be  regarded  as  inconclusive,  because 
of  the  simplicity  of  the  test  scene  and  the  reliance  on  simulated  range 
data  for  assigning  the  initial  interpretations.  Further  experiments 
in  more  complex  scenes  are  planned  using  an  actual  laser  range  finder. 
Additional  experiments  will  be  performed  without  range  data,  using 
constraints  that  infer  height  and  orientation  from  pictorial  cues  such 
as  image  height  and  shadows. 


A.  Introduction 

In  this  appendix,  we  establish  the  admissibility  and  optimality  of 
♦ 

algorithm  M , when  used  with  supportive  constraints.  We  will  first  establish 
conditions  of  admissibility  and  optimality  for  an  abstract  tree  searching 

algorithm  A that  encompasses  M and  then  show  that  with  supportive  constraints, 

* 

M fulfils  those  conditions. 

B . Definitions 

We  consider  the  following  search  problem:  Let  Tr(s)  be  a finite  tree 

with  root  node  s.  For  each  node  n in  Tr(s),  let  Tr(n)  be  the  sub-tree 

* 

with  root  node  at  n.  Each  tip  node,  r,  of  Tr(s)  has  a value  v(r)  . Let  r , 
with  value  v*,  be  called  a best  tip  node  if  no  other  tip  node  has  a larger 
value.  For  non-tip  nodes,  n,  let  v(n)  be  the  value  of  the  best  tip  node 
in  Tr(n) . We  note  that  for  all  n in  Tr(s),  v(n)  < v . 

We  are  concerned  with  search  processes  for  finding  a best  tip  node  in 

Tr(s)  . This  problem  is  analogous  to  that  of  finding  a least  costly  path 

$ 

in  a tree.  For  the  latter  problem  there  is  a search  algorithm,  A , for 
which  admissibility  and  optimality  theorems  have  been  proved.  Here  we 
prove  analogous  theorems  for  an  analogous  algorithm,  A+,  that  finds  best 
tip  nodes  . 

C . Algori t hm  A 

Let  v be  an  estimating  function  for  estimating  the  value  of  nodes  in 
Tr(s)  . That  is,  v(n)  is  an  estimate  of  the  value,  v(n),  of  the  best  tip 
node  in  Tr(n) . Search  algorithm  A+  is  defined  as  follows: 


1.  ^it  node  s on  OPEN. 

2.  select  that  node,  n,  on  OPEN  with  the  largest  value  of  v.  Resolve 
ties  arbitrarily  but  always  in  favor  of  tip  nodes. 

3.  If  node  n is  a tip  node,  terminate;  else  continue . 

4.  Expand  node  n by  putting  its  successors  on  OPEN.  Remove  node  n 
from  OPEN. 

5 . Go  to  2, 

+ The  collaboration  of  Nils  Nilsson  in  preparing  this  appendix  is  gratefully 
acknowledged  . 
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With  the  assumption  that  v(n)  & v(n)  for  all  n in  Tr(s),  and  that  v(r)  = v(r) 
for  all  tip  nodes,  r,  we  make  the  following  observations: 

A)  At  any  time  prior  to  termination  of  A , a best  tip  node  in 
Tr(s)  is  in  the+Tr(n)  for  some  node  n on  OPEN.  For  this  node  n, 

v(n)av(n)  v . 

B)  A never  re-opens  nodes,  because  it  is  searching  a tree. 

C)  Since  Tr(s)  is  finite,  A^  must  terminate.  We  note  that  it 
can  only  terminate  by  selecting  a tip  node  in  step  3. 

T 

D . Admissibility  of  A 

We  say  that  an  algorithm  is  admissible  if  it  terminates  by  finding 
a best  tip  node. 

Theorem  1 If  v(n)  ^ v(n)  for  all  nodes  n,  and  v(r)  = v(r)  for  tip 
nodes  r,  then  A is  admissible. 

Proof:  Assume  the  contrary.  That  is,  assume  that  A+  terminates  for 

/v  * 

some  tip  node  r'  that  is  not  best,  i.e.  for  which  v(r')  = v(r')  < v . But 
then  just  before  selecting  r ',  by  observation  A above,  there  was  a node 

a * + 

n on  OPEN  with  v(n)  > v contradicting  our  assumption  that  A chose  r 
instead  of  n. 

E . Optimality  of  A+ 

First  we  pr  ve  a lemma. 

Lemma:  If  A+  expands  a non-tip  node,  n,  and  if  v(n)  > v(n)  for  all  n, 

/s  * 

then  v(n)  a v . 

Proof:  By  observation  A,  at  the  time  A+  expanded  n,  there  existed  on 

OPEN  a node  n'  with  v(n')  & v . If  n ' = n,  v(n)  a v . Otherwise,  A 

A A * 

chose  n in  preference  to  n'  and  v(n)  a v(n')  a v . 

Theorem  2:  Let  two  versions  of  A , namely  A^  and  A.,  , search  a tree, 

Tr(s),  to  termination  using  estimating  functions  and  v^,  respectively. 

Then,  if  v (n)>v^(n)  a v(n)  for  all  non-tip  nodes  n in  Tr(s),  and  if 
v.,(r)  v^Cr)  = v(r)  for  all  tip  nodes  in  Tr(s),  A^  will  expand  all  the 
non-tip  nodes  expanded  by  A^+  . 

Proof:  Suppose  the  contrary.  Let  Tr, (s)  and  Tr_(s)  be  the  trees  of  nodes 

expanded  by  A^  and  A,/  , respectively.  If  Tr^s)  contains  a non-tip  node 

n«*t  contained  in  Trv(s),  then  (since  both  are  rooted  in  s)  there  will 


exist 

a non- 

•tip 

node,  n,  in  Tr,(s), 

not 

in  Tr., (s), 

with  a parent  in  both. 

+ 

That 

is,  at 

the 

termination  of  A , 

node 

n is  on  the  OPEN  list  of 

But  if 

V 

didn't  expand  node 

n,  it 

must  have 

& A . 

been  because  v • v,,(n) 

Also, 

since 

Ax+d 

id  expand,  n we  have  by 

the  Lemma 

that 

v1(n)  > v . 

But  these  relations  contradict  our  assumption  that  for  non-tip  nodes 
v,,  (n)  -»  v^(n)  . 

F.  Discussion 

A -f- 

Theorem  1 states  that  as  long  as  v is  an  upper  bound,  algorithm  A is 

admissible.  Clearly,  the  search  can  be  safely  terminated  when  a terminal 

node  is  encountered  whose  actual  score  is  higher  than  upper  bounds  on  the 

scores  of  all  terminal  nodes  that  are  reachable  from  any  open  node.  Theorem 

2 states  that  A+  is  optimal  in  the  sense  that  it  will  never  expand  more 

nodes  than  any  other  admissible  algorithm  that  relies  on  score  estimates 

that  are  strictly  larger  than  the  upper  bound  estimates  used  by  A . It 

* 

now  remains  to  be  shown  that  algorithm  M is  an  instance  of  the  more  general 

+ * 
algorithm  A and  that  with  supportive  constraints,  M fulfills  the  conditions 

required  for  proving  Theorems  1 and  2. 

* + 

M like  A is  an  algorithm  for  finding  the  highest  scoring  terminal 
node  in  a tree  . In  scene  analysis,  the  nodes  of  the  tree  represent  states 
of  instantiation  that  are  reached  by  pinning  the  likelihoods  of  particular 
region  interpretations  to  zero.  Nodes  representing  a state  where  every 
region  has  a unique  interpretation  are  designated  terminal  nodes.  Two 
arcs  emanate  from  each  non-terminal  node  representing  the  assertion  and 
denial  of  the  most  likely  interpretation  of  a previously  uninstantiated 
region . 

* 

M selects  for  expansion  the  node  with  the  highest  valued  score.  This 
score  is  formed  by  summing  the  highest  interpretation  likelihood  associated 
with  each  region.  Since  these  likelihoods  are  in  fact  upper  bound  estimates, 
the  score  of  a non-terminal  node  is  an  upper  bound  'in  the  score  of  any 
terminal  node  reachable  by  further  instantiation.  The  restriction  to 
supportive  constraints  means  that  interpretation  likelihoods  will  never 


A 


increase  as  a consequence  of  decreasing  the  likelihood  of  another  interpre- 
tation. In  particular,  since  instantiations  are  accomplished  by  setting 
the  likelihood  of  alternative  interpretations  to  zero,  they  cannot  raise 
the  likelihood  of  any  uninstantiated  interpretation.  Consequently,  the 
score  of  a node  is  a true  upper  bound  on  the  combined  interpretation 

likelihoods  at  any  terminal  node  accessible  from  that  node.  This  estab- 

* * 

lishes  the  admissibility  of  M . The  optimality  of  M follows  from  the 
fact  that  the  score  of  a node  is  at  least  as  tight  a bound  as  one  based 
on  fewer  supportive  constraints  or  restricted  propagation.  Both  of  these 
factors  can  only  raise  individual  likelihood  estimates,  thereby,  loosening 
the  upper  bound  on  score.  The  tighter  the  bound  on  score,  the  more 
directed  the  search. 


Appendix  B 


DETAILED  TRACE  OF  AN  INTERPRETATION 

This  appendix  contains  a detailed  trace  of  the  interpretation  process 
for  the  vertical  regions  in  Figure  4 as  described  in  Sections  II. C and  III. 
Jobs  executed  from  the  job  queue  are  preceded  by  the  designations  JOB  or  BG. 
BG  stands  for  background  jobs  that  were  either  placed  on  the  queue  to 
initialize  processing  (e.g.,  NETSETUP  jobs)  or  added  to  the  end  of  the  queue 
to  be  processed  last  (e.g.,  OPTION  variables).  All  other  reevaluations  are 
added  dynamically  to  the  front  of  the  queue  so  that  current  lines  of  deduc- 
tion are  pursued  first. 

The  real  valued  numbers  following  each  job  are  updated  likelihoods  of 
the  corresponding  XDEMON  variables  produced  by  running  the  job. 

A.  Initialization  Phase 

Lines  1 through  83  of  the  trace  (Figure  B-l ) constitute  the  initializa- 
tion phase  during  which  the  constraint  network  is  constructed  and  equili- 
brium likelihoods  are  obtained.  Each  NETSETUP  job  retrieves  the  constraints 
associated  with  an  interpretation  and  then  constructs  and  executes  a global 
evaluation  function  that  computes  support  for  those  constraints  in  the  cur- 
rent scene  partition. 

The  tag  (0.0-setup)  indicates  an  interpretation  that  was  rejected  out- 
rightly  because  of  an  unsupported  constraint.  The  likelihood  of  such  inter- 
pretations are  permanently  set  to  zero  and  no  evaluation  function  is  compiled 
for  them.  In  line  8,  for  example,  the  interpretation  (DOOR  CBACK)  was  re- 
jected by  the  constraint  ROOMPARTITION.  The  resultant  drop  in  likelihood 
of  (DOOR  CBACK)  triggered  reevaluation  of  the  likelihood  functions  for 
(DOOR  LWALL)  at  line  13  and  (DOOR  RWALL)  at  line  16  which  had  been  pre- 
viously evaluated  in  lines  4 and  6,  respectively,  using  the  a priori 
likelihood  of  (DOOR  CBACK).  (In  effect,  decreasing  the  likelihood  that 


a dark  region  was  DOOR  increased  the  likelihood  that  DOOR  was  a light 
region.)  Similarly,  the  likelihoods  of  (WALL  LWALL)  and  (WALL  RWALL)  are 
increased  at  lines  21  and  22  after  rejecting  (WALL  CBACK)  at  line  18. 
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00001  BG-  ( NETSETUP  (QUOTE  (WASTEBASKET  WBSKT ))) • .12 

00002  BG- (NET  SETUP  (QUOTE  (DOOR  WBSKT))):  0.0-SETUP 

00003  BG- ( NETSETUP  (QUOTE  (WALL  WBSKT))):  0.0-SETUP 

00004  BG- ( NETSETUP  (QUOTE  (DOOR  PWALL))):  .0686 

00005  BG- ( NETSETUP  (QUOTE  (WALL  PWALL))):  .0328 

00006  BG-(NETSETUP  (QUOTE  (DOOR  LW ALL  ) ) ) : . 0686 

00007  BG- ( NET  SETUP  (QUOTE  (WALL  L» ALL  ) ) ) I .0328 

00008  BG- ( NETSETUP  (QUOTE  (DOOR  CBACK  ) ) ) : 0.0-SETUP 

00009  JOB-((OR*  (DOOR  DP)  (DOOR  PtC)  (DOOR  CBACk)  (DOOR  WBSKT))):  .459 

0001  0 JOB  - ( (NOT*  (OR*  (DOOR  DR)  (DOOR  PIC)  (DOOR  CBACK)  (DOOR  WBSKT)))):  .541 

00011  JOB-( (AND*  (NOT*  (OR*  (DOOR  DP)  (DOOR  PIC)  (DOOP  CBACK)  (DOOR  WBSKT))) 

00012  (NOT*  (AND*  (ADJ  LWALL  PIC)  (PICTURE  PIC))))):  ,379 

0001  3 JOB-((DiinR  LWALL)):  .086 

00014  JOB-( (AND*  (NOT*  (OR*  (DOOR  DP)  (DOOR  PIC)  (DOOR  CBACK)  (DOOR  WBSKT))) 

00015  (NOT*  ( AND*  (ADJ  PWALL.  PIC)  (PICTURE  PIC)))1):  .379 

00016  JOB- ( ( DOOP  PWALL)):  .086 

00017  BG-(NETSETUP  (QUOTE  (CHAIPBACK  CBACK))):  .11 

00018  BG-(NETSETUP  (QUOTE  (WALL  CBACK))):  0.0-SETUP 

00019  JOB- ((CP*  (WALL  DP)  (WALL  PIC)  (WALL  CBACw)  (WALL  WBSKT))):  ,864 

00020  JOB - ( (NOT*  (OR*  (wALL  DR)  (wALL  PIC)  (WALL  CBACK)  (WALL  wBSKT)))):  .136 

00021  J H B — ( (WALL  LWALL)):  .105 

00022  JOb-( (wALL  PwALL)):  .105 

00023  BG-(NETSETUP  (QUOTE  (PICTURE  PIC))):  .251 

00024  BG-(NETSETUP  (QUOTE  (DOOP  PIC))):  0.0-SETUP 

00025  JOB- ( ( OP*  (DOOP  DR)  (DOOP  PIC)  (DOOP  CBACK)  (DOOP  WBSKT)));  .227 

00026  JOB-( (NOT*  (OP*  (DOOP  DR)  (DOOR  PIC)  (DOOp  CBACK)  (DOOP  WBSKT)))):  .773 

00027  JOB-( (AND*  (NOT*  (OH*  (DOOR  DR)  (DOOR  PIC)  (DOOR  CBACK)  (DOOP  WBSKT))) 

00028  (NOT*  (AND*  (ADJ  LWALL  PIC)  (PICTURE  PIC)))));  .541 

00029  JOB  - ( (POOP  LWALL)):  .123 

00010  JOB- ( (AND*  (ADJ  LwALL  PIC)  (DOOP  LwALL))):  .123 

00031  JOB- ( (AND*  (NOT*  (OR*  (DOOP  DP)  (DOOP  PIC)  (DOOR  CBACK)  (DOOP  wBSKT))) 

0003?  (NOT*  (AND*  (ADJ  PWALL  PIC)  (PICTURE  PIC))))):  .541 

00033  JOB- ( ( DOOP  PwALL)):  .123 

00034  JOB- ( ( AND*  (ADJ  RWALL  PIC)  (DOOP  HWALL))):  .123 

00035  BG-( NETSETUP  (QUOTE  (WALL  PIC))):  0.0-SETUF 

000)6  JOB- ( ( OR*  (wALL  DR)  (WALL  PIC)  (WALL  CBACK)  (WALL  WBSKT))):  .77) 

00037  JOP-( (NOT*  (OR*  (WALL  DR)  (wALL  PIC)  (WALL  CBACK)  (WALL  WBSKT))));  .227 

00038  JOB-( (wALL  LWALL)):  .175 

00039  JOB-KWALL  HWALL)):  .175 

00040  RG-(NETSETUP  (QUOTE  (DOOP  DP))):  .175 

00041  JUB-( (OP*  ( DCOP  DP)  (DOOR  PIC)  (DOOR  CBACK)  (DOOP  WBSKT))):  ,175 

0004?  JOB- ( ( NOT  * (OP*  (DOOP  DP)  (OODP  PIC)  (DOOP  CBACK)  (DOOP  wBSKT ))))  : .826 

00043  JOb-( (AND*  (NOT*  (OP*  (DOOR  DR)  (DOOR  PIC)  IDOOP  CBACK)  (DOOR  WBSKT))) 

00044  (NOT*  (AND*  (ADJ  LWALL  PIC)  (PICTURE  PIC))))):  .578 

00045  JOB-((AND»  (NOT*  (OP*  (DOOP  DR)  (DOOR  PIC)  (DOOP  CBACK)  (DOOP  WBSKT))) 

00046  (NOT*  (AND*  (ADJ  RWALL  PIC)  (PICTURE  PtC))))):  ,578 

00047  BG -(NET SETUP  (QUOTE  (WALL  DR))):  .526 

00048  JOB-( (OP*  (WALL  DP)  (WALL  PIC)  (WALL  CBACK)  (WALL  WBSKT))':  .526 

00049  JOB- ( ( NUT  * (OR*  (WALL  DR)  (WALL  PIC)  (WALL  CBACK)  (wALL  oSKT)))):  .474 

00050  JOB- ( (WALL  LwALL)):  .367 

00051  JOb-( (OR*  (WALL  LWALL)  (WALL  RwALL))):  .478 

FIGURE  B-1  TRACE  OF  EXECUTION  DURING  ESTABLISHMENT  OF  NITIAL  EQUILIBRIUM 
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00052 

00053 

00054 

00055 
0005b 
00057 
00  OS  8 
00059 
OOObO 
000b  1 
00  062 
00063 
000b4 

00065 

00066 
00067 
00  Ob  6 
00069 
00  070 
00071 
0007? 
0007  3 

00074 

00075 

00076 

00077 

00078 

00079 

00080 
00081 
0008? 
00083 


JOB-((NOT*  (OR*  (WALL  LWALL ) (WALL  RWALL)))):  .522 
JOB- ( (WALL  DR)  ) : .404 

JOB - ( (OR*  (WALL  DP)  ( vALL  PIC)  (WALL  CbACK)  (WALL  WBSK  T ) ) ) : .404 

J 0 B - ( (NOT*  (OP*  (WALL  DR)  (WALL  PIC)  (WALL  CBACK)  (wALL  WBSKT)))):  .596 

JDB-((WALL  LWALD):  .461 

JOB-((OP*  (WALL  LwALL)  (wALL  RWALL))):  .556 

JOB-((NOT*  (OR*  (WALL  LWALD  (WALL  P W ALL ) ) ) ) : .444 
JOB-((WALL  DR)):  .344 

JOB-((OR*  (WALL  DP)  (wALL  PIC)  (WALL  CBACK)  (WALL  WBSKT))):  .144 

J()R»( (NOT * (OR*  (wALL  DR)  (WALL  PIC)  (WALL  CBACK)  (WALL  WBSKT)))):  .656 

JOB  * ( (WALL  LWALD):  .507 
JOP-((wALL  RwALD):  .507 

JOb-( (OP*  (WALL  LWALL)  (wall  PWALL))):  .757 
JOB-((NUT*  (OH*  (WALL  LWALL)  (WALL  RWALL)))):  .243 
JOB* ( (WALL  DP  ) ) : .188 

JOB* ( (OP*  (wALL.  DR)  ( wALL  PTC)  (WALL  CBACK)  (wALL  WBSKT))):  .108 

JOB* ( (NOT*  (OR*  ( w ALT  DP)  (wALL  PIC)  (WALL  CBACK)  (WALL  WftSKT)))):  .012 

JOB - ( (WALL  LWALL)):  .b2B 

JOB-( (OR*  (WALL  LWALL)  (WALL  RWALL))):  .8)7 

JOB  - ( ( NUT  * (OR*  (wall  LwALL)  (wALL  PwALL)))):  .103 

JOB- ( (WALL  DR)  ) : .142 

JOB* ( (WALL  RWALL)):  .628 

JOB  - ( ( PP*  (WALL  LWALL)  (wALL  PWALL))):  .862 

pr,-(  (OPTION  (DOOR  DR)  (wALL  DR)));  (DOOR  DP)-. 175 

BG- ( (OPTION  (PICTURE  PIC)  (DOOP  PIC)  (WALL  PIC))):  (PICTURE  PIC)-. 251 
BG- ( (OPTION  (DOUR  CBACK)  (CHAIRBACK  CBACK)  (WALL  CBACK))):  (CHAIPBACK 
CBACK  )-. 1 1 

R G • ( (OPTION  (DOOR  LwALL)  (WALL  LWALD)):  (WALL  LWALL)-. 628 
BG-( (OPTION  (DOOR  RWALL)  (WAIL  RWALL))):  (WALL  RWALL)-, 628 
BG-( (OPTION  (WASTEBASKET  WBSKT)  (DOOR  wBSKT)  (WALL  WBSKT))):  ( 
►ASTEBASKtT  WBSKT)-. 12 

SAVLSTATE:  (TOPCNTXT1,  SCORE : 1.91,  LENGTH ( 1 QUEUE  ) : 6 


FIGURE  B-1  TRACE  OF  EXECUTION  DURING  ESTABLISHMENT  OF  INITIAL 
EQUILIBRIUM  (Concluded) 
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Actual  computation  of  the  equilibrium  likelihoods  shown  in  Figure  5 
is  completed  by  line  74  following  execution  of  all  NETSETUP  jobs  and 
consequent  reevaluations.  This  phase  is  followed  by  the  execution  of 
OPT  ION  variables  in  lines  75  through  82,  which  check  for  contradictions 
(i.e.,  a region  left  without  a likely  interpretation),  increment  the 
global  score,  and  insert  pointers  to  themselves  in  the  priority  queue 
(IQUEUE)  governing  instantiation.  OPTION  variables  return  as  a value, 
the  likelihood  of  the  most  probable  interpretation  for  their  associated 
region.  The  name  of  the  interpretation  is  printed  along  with  this  likeli- 
hood for  experimental  convenience.  The  initialization  process  is  terminated 
in  line  83  by  saving  the  resultant  state  and  XDEMON  variables  before  any 
instantiation.  The  search  state  is  summarized  by  printing  the  summed 
likelihoods  of  the  current  best  interpretation  for  every  region  (SCORE) 
and  the  number  of  instantiated  regions  remaining  in  that  state  and  repre- 
sented on  IQUEUE. 

B.  Search  Phase 

In  this  phase,  MSYS  performs  a best  first  search  for  the  set  of 
unique  region  interpretations  with  the  highest  combined  likelihood.  (See 
Figure  B-2. ) The  search  begins  at  line  84  by  reinstating  the  highest 
scoring  state  from  SQUEUE  which,  by  default,  is  TOPCNTEXT.  The  most 
likely  interpretation  of  an  uninstantiated  region  (WALL  RWALL)  is  then 
instantiated  in  line  85.  The  instantiation  sets  to  zero  the  likelihoods 
of  alternative  interpretations  of  the  region  RWALL.  These  adjustments 
trigger  reevaluations  (lines  86  through  92)  which  raise  the  likelihood 
of  (DOOR  DR). 

State  variables  are  then  updated  at  line  93  by  executing  the  OPTION 
variable  for  region  RWALL,  the  only  region  experiencing  a significant 
(greater  than  0.05)  change  in  an  interpretation  likelihood.  Since  inter- 
pretation ambiguities  remain  but  no  contradictions  were  detected,  the 
resulting  search  state  is  saved  on  SQUEUE  (line  94),  labeled  by  its 
instantiation  history.  The  alternative  state  based  on  denying  (WALL  RWALL) 
is  evaluated  in  lines  96  to  112  and  placed  on  SQUEUE  (line  113)  behind  the 
higher  scoring  state,  (((WALL  RWALL)  1.0)  TOPCNTXT) . This  concludes  the 
first  stage  of  search. 
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00  0 A 4 PtlNSTATE:  (TOPCNTXT) 

00085  ASSFPT  - (PALL  RWALL) 

00086  JOB-((UP*  (DOOR  LwALL)  (DOOP  PWALL))):  .123 

000P7  JOB-((NOT*  (OP*  (DOOR  LWALL)  (DOOR  PW ALL  1)73:  .877 

00088  JOB-((AND»  (NOT*  (OP*  (DOOP  LWALI  ) (DOOR  RWALL ) ) ) (NOT*  0.0))):  .«77 

00088  JOB  - ( ( DOOP  DP)):  ,198 

00090  J|)R-((AND«  (ADJ  RwALI.  PIC)  (DOOP  RWALL)  ) ) ; 0.0 

00091  JI1H-(((IP*  (AND*  (ADJ  P 8 ALL  PIC)  (DOOR  RWALL))  (AND*  (ADJ  LWALL  PIC) 

00092  (DOOP  LWALL  ))))  : .123 

00093  BG- ( ( OPT  I ON  (DOOR  RWALL)  ( WALL  PWAI.L  ) ) ) : (WALL  PWALL)-. e28 

00094  SAVESTATI:  ((('WALL  RWALL)  . .b29)  TOPCNTXT),  SCOPE:  1.91 

00095  , LENGTH ( IQUEUE ) : 5 

00096  REINSTATE:  (TOPCNTXT) 

00097  DENY  - (WALL  PwALL) 

00098  JOB • ( ( OP*  (WALL  LWALL ) (WALL  RWALL))):  .628 

00094  J O B - ( (NOT*  (OP*  (WALL  LWALL)  (WALL  PWALL)))):  .372 

00100  JOB  - ( (WALL  DR)):  .288 

00101  JOB” ( (OP*  (WALL  DP)  (WALL  PIC)  (WALL  CBACK)  (wALL  WbSKTjl):  .288 

00102  J()b« ( ( NOT  * (OR*  (wALL  DP)  (WALL  PIC)  (WALL  CBACK)  (WAIL  WBSKT ) ) ) ) : .712 

00103  JOB- ( ( "ALL  LWALL)):  .551 

00104  JOB  - ( (OR*  ("ALl  LwALL ) (wALL  RwALL))):  .551 

001  05  JOB - ( (NOT * (OR*  (WALL  LwALL)  (WALL  PwALL)))):  .449 

00106  JOB-< (WALL  DP)):  .347 

00107  JOB-  ( ( (JP  * (WALL  DP)  (WALL  PIC)  (WALL  CBACK)  (WALI.  WBSKT)))j  .347 

00108  JOB - ( ( NOT  * (OR*  (wALL  DR)  (WALL  PIC)  (WALL  CBACK)  (WALL  WBSKT ) ) ) ) : .653 

00109  JOB- ( ( WALL  LWALL)):  .505 

00110  pr.-((OPTins  (DOOP  RWALL)  (WALL  PWALL))):  (DOOR  PWALL)-, 123 

00111  PC, -(  (OPTION  (LOOP  DP)  (WALL  DP)));  (WALL  DR)-. 347 

00112  PG-  ( (OPTI UN  (DOOP  LwALL)  (WALL  LWALL))):  (WALL  LWALI)-. 505 

00113  SAVESTATE:  (((WALL  PWALL)  . 0)  rOPCNTXT),  SCOPE:  1.46,  LENGTH ( I OJEUE ) : 6 

00114  PEINfTATF : (((WALL  PWALL)  . .628)  TOPCNTXT) 

00115  ASSERT  - (WALL  LWALL) 

00116  J()P-((OP*  (POUR  LWALL)  (DOOR  PwALL))):  0.0 

00117  JOB - ( ( NOT  * (OP*  (DOOR  LWALL)  (DOOP  RWALL)))):  1.0 

00118  JOB-( (AND*  (NOT*  (OP*  (DOOR  LWALL)  (OUOP  pwAI.L)))  (NOT*  0.0))):  1.0 

00119  JOB- ( (DOOR  DP)):  .227 

00120  JOB-( ( AND*  (ADJ  LWALL  PIC)  (DOOR  LwALL))):  0.0 

00121  JOB - ( ( OR*  (AND*  (ADJ  RWALL  PIC)  (DOOR  PwALL))  (AND*  (ADJ  LWALL  PIC) 

00122  (HOOP  IwaLL)))):  0.0 

00123  JOB- ( ( NOT*  (OP*  (AND*  (ADJ  RWALL  PIC)  (DOOP  RWALL))  (AND*  (ADJ  LWALL 

00124  PIC)  (DOOR  LWALL))))):  1.0 

0012R  JOB-( (PICTURE  PIC)):  .3 

00126  HO- ((OPTION  (DOOR  LWALL)  (WALL  LWALI))):  (WALL  LWALL)-. 628 

00127  RG- ( (OPTI ON  ( DOOR  DP)  (WALL  DP)));  (DOOP  DR)-. 227 

00  128  SAVESTATF : (((WALL  (.WALL)  . .629  ) ((WALL  RWALL)  . .628  ) TOPCNTXT) 

00129  , SCORb  : 1.9ft,  I,ENGTH(  IQUEUO)  1 4 

001  30  PE  INSTATE : (((WALL  RWALI)  . .628  ) TOPCNTXT) 

001 31  TENY  - (WALI  LWALL) 

00132  JOB-((PH*  (wALL  LwALL)  (WAIL  PWALL))):  .629 

001  33  JOB - ( (NOT * (OP*  (WALL  LwALL)  (WALL  PwALL)))):  .372 

00134  JOB- ( ( WALI  DP)):  .288 


FIGURE  B-2  CONTINUATION  OF  TRACE  SHOWING  EXECUTION  OF  SEARCH 
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00135  jnb-((OR*  (WALL  DP)  (WALL  PIC)  (WAIL  CBACK)  (WALL  WBSKT )))  : .288 

00136  JOB - ( (NOT*  (OP»  (WALL  DR)  (WALL  PIC)  (WALL  CBACK)  (WALL  WBSKT))))!  .71? 

001  37  Jl)b  - ( (WALL  RWALL)):  .551 

00 1 3w  jnp.((CiR#  (WALL  LWALL)  (WALL  RWALL)))!  .551 

00139  jnB-((NOT*  (OP*  (WALL  LWALL)  (WALL  RWALL))))!  .449 

00140  JOB-((WALL  DP)):  .347 

00141  JOP-((OR*  (wALL  DR)  (wALL  PIC)  (WALL  CBACK)  (WALL  wBSKT)))!  .347 

00142  JOB - ( (NOT*  (OR*  (wALL  DR)  (wALL  PIC)  (WALL  CBACK)  (WALL  WBSKT))))!  .653 

00143  JOB- ( ( wALL  RWALL))!  .505 

00144  BG-((UFTTON  (POOR  LWALL)  (WALL  LWALL)))!  (DOOR  LWALL)-, 123 

001  45  bG-( (OPTION  (DOOR  DR)  (wALL  DR))).  (WALL  DP)-. 317 

0014B  RG-( (OPTION  (DOOR  RWALL)  (WALL  RWALL))):  (WALL  PWALL)-,505 

00147  SAVESTATE;  (((WALL  LWALL)  . 0)  ((WALL  RwALL)  , .629)  TOPCNTXT),  SCOPE:  ).46 

0014R  , LENGTH ( IQUEUE) : 5 

00149  REINSTATE:  ((("ALL  LwALL ) . .628)  ((WALL  PWALL)  . .628)  TOPCNTXT) 

00150  ASSEPT  - (PICTURE  PIC) 

00151  SAVESTATE:  (((PICTURE  PIC)  . .3)  ((WALL  LwALL)  . .628)  ((WALL  RwALL) 

00152  . .62«)  TOPCNTXT),  SCOPE:  1.96,  LENGTH  ( I QUEUE* ) : 3 

00153  PE INSTATE ; (((wALL  LWALL)  . .629)  ((WALL  FWALL ) . .628)  TOPCNTXT) 

00154  TEN  V - (PICTURE  PIC) 

001  55  R G - ( (OPTION  (PICTURE  PIC)  (DOOR  PIC)  (WALL  PlC)))i 

00156  "CONTRADICTION  DETECTED" 

00157  REINSTATE:  (((PICTURE  PIC)  . .3)  ((WALL  LWALL)  . .629)  ((WALL  RWALL) 

00 1 S 8 . .628  ) TOPCNTXT) 

00159  ASSERT  - (POOR  DR) 

00160  JOB - ( (OR*  (wALL  DP)  (wALL  PIC)  (WALL  C RACK)  (WALL  WBSKT)))i  0.0 

00161  JOB - ( (NOT*  (OR*  (wALL  DR)  (WALL  PIC)  (WALL  CRACK)  (WALL  WBSKT)))):  1.0 

00162  J 0 B • ( ( WALL  LwALL))!  .773 

00163  JOR-( (WALL  RWALL))!  .771 

00164  RG- ( (OPTION  (DOOR  DR)  (WALL  DR)))-  (DOOR  OP)-. 227 

00165  p.i-(  (OPTION  (DOOR  LwALL)  (WALL  LWALL)))!  (WALL  LWALL)-. 773 

00166  b G • ( (OPTION  (DOOR  RwALL)  (WALL  RW4LL)))!  (WALL  RWALL)-. 773 

00167  SAVESTATE:  (((DOOR  DR)  . .227)  ((PICTURE  PIC)  . .3)  ((WALL  LWALL) 

OOlhO  . .629)  ((WALL  RwALL)  . .629  ) TOPCNTXT),  SCUREl  2.25,  LENGTH ( I QUEUE ) : 2 

00169  REINSTATE!  (((PICTURE  PIC)  . .3)  ((wALL  LWALL)  . .628)  ((WALL  RWALL.) 

00170  , .628)  TOPCNTXT) 

00171  DENY  - (DOOR  DR) 

0017?  RG-( (OPTION  (DOOR  DP)  (WALL  DR))):  (WALL  DP)-. 142 

00173  SAVESTATE-  (((DOOR  DP)  . 0)  ((PICTURE  PIC)  . .3)  ((WALL  LWALL)  . .628) 

001  74  ((WALL  RwALL)  . .628  ) TOPCNTXT),  SCORE:  l.RP,  LENGTH ( IQUEUE ) | 3 
00175  REINSTATE:  (((DOOR  DF)  . .227)  ((PICTURE  PIC)  . .3)  ((WALL  LWALL) 

00(76  . .628)  ((WALL  RWALL)  . .628)  TOPCNTXT) 

00177  ASSERT  - (WASTEBASKET  WBSKT) 

00178  SAVESTATE!  (((WASTEBASKET  WBSKT)  . .12)  ((DOOR  Dp)  . .227)  ((PICTURE 

0017R  PIC)  . .3)  ((WALL  LWALL)  . .628  ) ((WALL  RwALL)  . .628  ) TOPCNTX!  ) 

00180  , SCORE:  2.25,  LENGTH ( I QUEUE ) I 1 

00181  REINSTATE!  (((CHOP  DP)  . .227)  ((PICTURE  HC)  . .3)  ((WALL  LWALL) 

00182  . .629)  ((WALL  PWALL)  . .628)  TOPCNTXT) 

00183  DENY  - (WASTEBASKET  WBSKT) 

001H4  BG- ( ( OFT  t ON  (WASTEBASKET  WBSKT)  (DOOR  WBSKT)  (WALL  wBSKT))): 

00185  "CONTRADICTION  DETECTED" 

00186  REINSTATE:  ( ( ( W ASTF B ASK ET  WBSKT)  . .12)  ((DOOR  DR)  . .227)  ((PICTURE 

00187  PIC)  . .3)  ((WALL  LWALL)  . .628)  ((WALL  PWALL)  , .628)  TOPCNTXT) 

00188  ASSERT  - (CHAIRBACK  CBACK) 

00189  "SUCCESS" 


FIGURE  B-2 


CONTINUATION  OF  TRACE  SHOWING  EXECUTION  OF  SEARCH 


(Concluded) 


Subsequent  stages  of  search  proceed  in  an  analogous  manner,  by  rein- 
stating the  top  scoring  state  from  SQUEUE  and  then  exploring  the  consequences 


of  asserting  and  denying  the  highest  likelihood  interpretation  of  an  unin- 
stantiated region.  The  next  step,  for  example,  explores  the  consequences 
of  asserting  and  denying  the  hypothesis  (WALL  LWALL)  in  the  restored  state 
(((WALL  RWALL)  1.0)  TOPCNTXT)  (lines  114  to  148).  In  this  example,  the 
search  proceeded  directly  to  the  desired  global  scene  interpretation,  guided 
by  monotonically  increasing  state  scores  resulting  from  a sequence  of  correct 
instantiations. 

Note  the  decreased  amount  of  propagation  following  an  instantiation 
and  the  detection  of  global  inconsistency  (lines  155  and  184)  as  the  search 
becomes  progressively  more  constrained.  The  search  terminates  successfully 
at  line  189  with  IQUEUE  empty  and  no  region  without  interpretation.  Alter- 
native sets  of  consistent  interpretations,  should  any  exist,  could  be 
developed  in  order  of  decreasing  goodness  by  continuing  to  search  remaining 
contexts  until  SQUEUE  was  emptied. 

C . Final  State  of  Data  Base  Following  Search 

The  final  state  of  the  data  base  following  search  is  shown  in  Figure 
13-3.  It  differs  from  the  initial  equilibrium  state  (Figure  8)  in  two  ways: 
First,  the  likelihoods  of  incorrect  interpretations  have  been  reduced  to 
zero.  Second,  the  relative  lists  of  some  variables  (e.g.,  (DOOR  DR))  have 
been  pruned  by  removing  other  variables  whose  likelihoods  are  pinned  at 
zero  or  whose  own  relative  lists  have  become  empty. 
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00001 

VARIABLE: 

(WASTEBASKET  wBSKT) 

00002 

VALUE : 

.12 

00003 

PROCFDUPF • 

00004 

(WASTEBASKET  WBSKT) 

00005 

RELA1 IVES : 

0000b 

((OPTION  (WASTEBASKET 

NBSKT ) (DOOR  NBSKT ) (WALL  wBSKT ) ) ) 

00007 

00008 

00008 

VARIABLE: 

(DOOR  WBSKT) 

00010 

VAl UF: 

0.0 

0001  1 

PROCEDURE : 

00012 

(DOOR  WBSKT) 

0001  3 

PEI  ATI  Vt.S  : 

000  1 4 

((OPTION  (WASTEBASKET 

WBSKT ) (DOOP  WBSKT)  (WALL  WBSKT))) 

00015 

0001b 

0001  7 

VARIABLE: 

(WALL  wbSKT) 

00018 

VALUE : 

0.0 

00018 

PROCFDUPE : 

00070 

(wALL  NBSKT) 

00021 

RELATIVES: 

00022 

((OPTION  (WASTEBASKET 

WBSKT)  (DOOR  WBSKT)  (WALL  WbSK  T ) ) ) 

00023 

( (OR*  ( WALL  DP ) ( WAI L 

PIC)  (WALL  CBACK)  (WALL  WBSKT))1 

00024 

00025 

0 0 0 2 6 

VARIABLE: 

(DOOR  RWALL1 

00077 

VALUE.; 

0.0 

00O2« 

FROCFOLRE • 

00078 

[AND*  .227  (ANO*  (NOT* 

(OR*  (Door  dp) 

000  30 

(Door  PIC) 

000  31 

(DOOR  CBACK) 

000  32 

(DOOR  WBSKT))) 

00033 

(NOT* 

(ANO*  (APJ  RWALL  PIC) 

000  34 

(PICTURE  PIC) 

00035 

RELATIVES; 

00036 

( (OPTION  (DOOR  RWALL) 

(WALL  RWALL))) 

00037 

((AND*  (ADJ  pwALL  PIC) 

(DOOR  PwALL))) 

000  38 

((DP*  (DO0P  L'WALL)  (DOOR  PWALL))) 

000  38 

00040 

00041 

VARIABLE: 

(WALL  RWALL) 

00042 

VAL  UF : 

.773 

0004  1 

PROCEDURF : 

00044 

(AND*  .773  (NOT*  (OP* 

(WALL  DR) 

00045 

(WALL  PIC) 

00046 

(WALL  CBACK) 

00047 

(WALL  wBSKT) 

00048 

RELATIVES: 

00048 

( ( OP 1 I ON  (DOOR  RWALL) 

(WALL  RwAI  L) ) ) 

00050 

00051 

FIGURE  B-3  DATABASE  FOLLOWING  SEARCH 
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00052 

VARIABLE: 

(DOUR  LWALL) 

00051 

VAj  UF : 

0.0 

00054 

00055 

PPOCFiKIRF: 

[AND*  .227  (AND*  (NOT* 

(OR*  (Door  dp) 

0005b 

(DOOR  PIC) 

0005  7 

( door  cback) 

0 005  R 

(Door  wpskt))) 

00050 

(NOT* 

(AND*  (ADJ  LWALL 

PIC) 

00060 

(PICTURE  PIC) 

00061 

00062 

RELATIVES: 

((OPTION  (DOOR  LwALL) 

(WALL  LWALL))) 

000b  1 

((AND*  (ADJ  LWALL  PIC) 

(DOOR  LWALL))) 

00064 

({OR*  (DOOR  LWALL)  (DOOR  RW&LL))) 

00065 

0006b 

00067 

VARIARI E: 

(WALL  LwAL  L ) 

0006R 

VALUE: 

.773 

0006  0 
0 0 0 7 0 

PROCEDURE: 

[AND*  .773  (NUT*  (OR* 

(WALL  DP) 

0007  1 

(wALL  PIC) 

00072 

(WALL  CRACK) 

0007  1 

(WALL  WPSKT) 

00074 

00075 

00076 

00077 

rflati vfs : 

( (UPTIUN  (DOOR  LaAlL  ) 

(WALL  LWALL) ) ) 

0 007  fl 

VARIABLE: 

(DOOR  CRACK  ) 

00070 

vat  Uf  : 

0.0 

00080 

PROCEDURE : 

00  0 R 1 
00082 

RELATIVES: 

(POUR  CBACK  ) 

OOOfl  3 

((OPTION  (OOOP  C8ACK) 

(CHATPflACK  CBACK) 

( WALL 

CBACK) ) ) 

000  R 4 
000  R 5 
00086 

VARIABLE : 

(CRAIP9ACK  CBACK) 

00087 

VALHF : 

.11 

00088 

00088 

PROCEDURE: 

(CHAIPBACK  CBACK) 

00080 
0008  1 
00092 

PHATIVES: 

( (OPTION  (DOOR  CBACK) 

(CHAIPBACK  CBACK) 

(WALL 

CBACK ) ) ) 

0008  1 
00084 

VARIABLE: 

(WALL  CBACK) 

00085 

VALUE: 

0.0 

00086 

00087 

PROCEDURE : 

(WALL  CBACK) 

00  098 
00088 

PFt  ATI VES : 

f (OPTION  (DOUR  CBACK) 

(CHAIRBACK  CRACK) 

(WALL 

CBACK) ) ) 

00100 

00101 
00102 

({OP*  (WALL  DR)  (WALL 

PIC)  (WALL  CBACK) 

(WALL 

WBSKT ) ) ) 

FIGURE  B-3 
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00103 

VAFIARL,  F : 

(PICTURE  PIC) 

00104 

VALUE: 

.3 

00105 

PROCEDURE; 

00106 

(AND*  .3  (HOT*  (OR*  (AMD#  (ADJ  PWALL  PIC) 

00107 

(DOOR  RWA1  L ) ) 

00108 

(AND*  (ADJ  LwALL  PIC) 

00109 

(DOOP  IiWALL  ] 

001)0 

RELATIVES: 

ooi  n 
00112 
00113 

((OPTION  (PICTURE  PIC)  (DOOP  PIC)  (WALL  PIC))) 

00114 

VARIABLE: 

(DOUR  PIC) 

00115 

VALUE: 

0.0 

00116 

PROCEDURE : 

00117 

(DOOR  PIC) 

001  1» 

RELATIVES: 

0011  4 
00120 
00121 

((OPTION  (PICTURE  PIC)  (DOOR  PIC)  (WALL  PIC))) 

00122 

VARIABLE: 

(WALL  PIC) 

00123 

VALUE: 

0.0 

00124 

PROCEDURE  : 

00125 

(WALL  PIC) 

0012b 

RELATIVES: 

00127 

((OPTION  (PICTURE  PIC)  (DOOR  PIC)  (WALL  PIC))) 

00128 
00129 
001  30 

((OP*  (WALL  OH)  (WALL  PIC)  (WALL  CRACK)  (WALL 

00131 

VARIABLE: 

(DOOF  DR) 

001  32 

VALUE  : 

.227 

00133 

PROCF  1>URE  ; 

00134 

(AND*  .227  (AND*  (NOT*  (OR*  (DOOR  LWALL) 

001  35 

(DOOR  PWALL))) 

001  3b 

(NOT*  0.0))) 

00137 

RELATIVES: 

001  3R 
001  39 
001  10 

((OPTION  (DOOP  DR)  (WALL  UP))) 

00141 

VARIABLE: 

(WALL  DP) 

00142 

VALUE : 

0.0 

00143 

PROCEDURE : 

00144 

(AND*  .773  (NOT*  (OP*  (WALL  LWALL) 

00145 

(WALL  PWALL) 

00146 

RELATIVES: 

00147 

((OPTION  (DOOP  DP)  (WALL  DR ) ) ) 

00148 
001  19 
00150 

((OP*  («ALL  DP)  (WALL  PIC)  (WALL  CBACK)  (WALL 

00151 

1 00 P 8 conses 

00152 

128*  second* 

FIGURE  B-3  DATABASE  FOLLOWING  SEARCH  (Concluded) 

Appendix  C 

REPRESENTATIONS  FOR  SPATIAL  RELATIONS 


Spatial  context  is  an  important  factor  in  resolving  interpretation 
ambiguities.  Procedural  representations  have  been  implemented  for  some 
common  three  dimensional  spatial  relationships  between  two  regions,  based 
on  the  relative  world  coordinates  of  vertices  in  their  polygonal  boundaries. 
These  representations  are  described  in  Table  C-l  and  demonstrated  in  Table 
C-2  using  the  test  regions  in  Figure  C-l . 

The  above  representations  were  originally  developed  for  room  scenes, 
assuming  availability  of  range  data.  All  relations  except  planarity  can 
be  reformulated  in  terms  of  two-dimensional  image  coordinates  for  standard 
eye  level  views. 


Table  C-2 

RELATIONS  OK  SURFACES  T'*  FrGUPF  <7  f'Sl'lC  RKppr  S K ►•'T  * T T P J S IN  TANJ.K  ?.t* 
(Table  lists  relations  of  ob lect  1 to  oblect  ?) 

where  a 3 above  F = J o front  L = left 

BL  a below  bk  = 1"  back  P = rloot 
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•iptfct  1 
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P 1 e t u r » 
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A ,PK 
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... 

P . BL 

P.A 
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I .A 
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1, 

L,» 

... 
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L,BL 
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L , Bl 

L,«J 

L , B L , F 

... 

of  determining  which  interpretations  of  the  selected  region  were  compatible 
with  the  other  interpretations  on  the  path  to  that  node.  Waltz  and  M*.  by 


Appendix  D 

XDEMON : A CONSTRAINT  SATISFIER 

A.  Overview 

A constraint  satisfaction  system  has  been  implemented  utilizing  coopei — 
ating  independent  processes  coupled  through  a global  data  base.  The  data 
base  consists  of  variables  representing  constrained  entities  and  constraints. 
Associated  with  each  variable  is  a procedure  for  computing  a value  in  terms 
of  the  current  values  of  other  variables.  Each  variable  also  has  a list  of 
related  variables  whose  procedures  utilize  the  present  variable  as  input. 

When  the  value  of  a variable  is  changed,  its  related  variables  are  activated 
by  adding  their  procedures  to  a queue  of  jobs  to  be  run.  Thus,  if  running 
a process  changes  the  value  of  its  associated  variable,  additional  processes 
may  be  activated.  Execution  terminates  when  the  job  queue  is  empty. 

B.  Details 

Each  variable  in  the  data  base  contains  four  components: 

. A FORM  (procedure  in  the  XDEMON  program) 

. A VALUE 

• A set  of  RELATIVES  (a  list  of  related  variables) 

. VARPROPS  (property  list). 

There  are  selecting  and  updating  functions  for  each  of  the  above 
components.  The  form  of  a variable  is  an  internal  representation  of  some 
s-expression . This  internal  representation,  known  as  an  H-expression,  is 
composed  of  either  a LISP  atom,  or  a list  of  other  variables.  Each  variable 
corresponds  to  a particular  s-expression:  in  LISP  a variable  corresponds 

to  a particular  atom,  and  we  have  generalized  this  notion.  For  example, 

(F00  X Y)  has  a corresponding  variable,  as  do  F00,  X,  and  Y.  So  also  does 
(FIE  (F00  X Y)  Z) . As  in  LISP,  character  strings  are  normalized  to  yield 
a unique  internal  representation — the  variable. 


( NORMEXPR  (expression))  is  a function  that  returns  the  variable 


Variables  are  initialized  to  have  the  value  Undef.  Relatives  and 
varprops  are  initialized  to  the  value  Nil. 

A collection  of  variables  may  be  linked  to  form  a network.  (CONNECT 
(variables)  puts  (variable''  on  the  lists  of  relatives  of  all  the  variables 
in  its  form.  (CONNECT!  (variable))  does  the  same  thing  recursively  for  the 
variables  in  the  form  as  well.  An  example  is  given  in  Figure  D-l . There 
are  corresponding  inverse  functions  DISCONNECT  and  DISCONNECT! . 

Note  that  pointers  to  subexpressions  are  available  via  the  form  and 
pointers  to  superexpressions  via  the  relatives. 

The  value  of  a variable  is  normally  set  by  the  function  HSET.  (HSET 
(variable)  ( value)) returns  (value)  as  its  result.  HSET  is  executed  for 
its  side-effects:  if  the  new  value  is  the  same  as  the  old  value  (under 

the  equivalence  HSETEQ,  initially  EQUAL),  nothing  happens:  if  the  new 

value  is  different,  then  the  variable's  relatives  are  evaluated  (or  rather 
the  evaluations  are  added  to  a list  of  jobs  to  be  run)  . Note 
that  other  schemes  could  be  used  here;  e.g. , some  relatives  might  be 
evaluated  before  the  variable  is  reset. 


FIGURE  D-1  REPRESENTATION  FOR  THE  EXPRESSION 
(CONNECT! 

(NORMEXPR 

(QUOTE 

(ANDIOR  A B) 

(OR  B (EQ  A C)))H) 

D-2 


Variables  are  evaluated  by  (HEVAL  (variable)).  HEVAL  evaluates  the 
form  of  the  variable,  and  then  HSETS  the  variable  to  the  new  value,  perhaps 
causing  its  relatives  to  be  evaluated.  It  returns  the  new  value  as  its 
result.  The  value  of  an  atomic  H-expression  is  the  value  of  the  variable: 
otherwise,  the  value  is  the  result  of  APPLYING  the  value  of  the  CAR  of  the 
H-expression  to  the  CDR. 

Unlike  LISP  the  definition  of  a function  is  kept  in  the  value  of  a 
corresponding  atomic  variable,  not  in  a special  cell.  Definitions  are 
established  be  executing  (HDEF  (function  definition)).  The  (function 
definition)  is  a list  of  two  elements,  the  name  and  the  body  (similar  to 
LISP's  DEFINE).  The  body  can  be  the  name  of  a LISP  function,  or  a lambda 
expression.  Note  also  that  when  a variable  is  HEVALuated,  the  evaluation 
is  not  recursive;  the  immediate  value  of  the  variables  forming  the  H- 
expression  are  utilized  without  HEVALuating  them. 

Variables  are  only  evaluated  when  the  value  of  something  to  which 
they  are  relatives  is  changed,  never  just  because  a variable  higher  in 
the  expression  is  evaluated.  Values  are  thus  remembered  and  not  recomputed 
unnecessari ly . 

A list  of  outstanding  jobs  to  be  run  is  held  in  the  global  variable 
JOBLIST.  Those  jobs  are  executed  by  a call  (RUNJOBS),  which  will  succes- 
sively execute  and  then  delete  the  jobs  on  the  list.  RUNJOBS  terminates 
when  there  are  no  jobs  left  (including  those  which  have  been  added 
dynamically ) . 

XDEMON  listings  are  available  from  the  authors. 
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AN  MSYS  EMIILAT  ION  OK  WALTZ  II  1,11  HI  NO 


*► 


«•> 

s 
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MSYS  can  be  easily  modified  to  emulate  .1  variety  of  search  paradigm- 
other  than  M* . This  appendix  describes  an  MSYS  emulation  of  W.ilt/.'- 
filtering  algorithm17  and  its  application  to  region  analysis. 

Waltz  analyzed  line  drawings  by  initially  assigning  all  locally 
possible  interpretations  to  each  vertex  and  then  eliminating  any  vertex 
interpretation  that  was  inconsistent  with  all  possible  interpretations 
of  a neighboring  vertex  along  a common  edge.  Eliminating  a possible 
vertex  interpretation  could  result  in  the  elimination  of  additional 
interpretations  from  adjacent  vertices.  This  elimination  process  would 
often  propagate  until  each  vertex  was  left  with  a unique  interpretation. 

A similar  paradigm  can  be  applied  to  region  analysis  by  initially  assign- 
ing all  locally  possible  interpretations  to  each  region  and  then  eliminat- 
ing interpretations  inconsistent  with  those  assigned  to  neighboring  regions 
sharing  a common  boundary.  Here,  inconsistency  is  defined  in  terms  of  a 
list  of  legally  adjacent  interpretations.  Such  a paradigm  was  implemented 
in  MSYS  and  used  to  analyze  the  scene  partition  of  Figure  4. 

The  analysis  utilized  the  adjacency  constraints  given  in  Figure  E-l, 
where  (LEGALADJ  II  (12  ...  IN))  specified  a list  (12  ...  IN)  of  legal 
interpretations  for  regions  adjacent  to  a region  with  interpretation  11. 

The  constraint  (LEGALADJ  DOOR  (DOOR  WALL  FLOOR  WASTEBASKET  TABLETOP)), 
for  example,  required  that  regions  labeled  DOOR  could  be  adjacent  only  to 
regions  labeled  DOOR , WALL,  FLOOR,  TABLETOP,  or  WASTEBASKET.  (In  a refine- 
ment on  Waltz,  it  was  further  required  that  two  door  interpretations  could 
be  legally  adjacent  only  if  the  regions  involved  had  similar  brightness.) 

WALL  regions,  similarly,  could  be  adjacent  only  to  regions  labeled  DOOR, 

WALL,  PICTITIE,  WASTEBASKET , or  TABLETOP.  These  adjacency  constraints  are 
somewhat  contrived  because  region  adjacency  is  an  ill-defined  concept  in 
a partially  partitioned  scene.  Fixtures,  such  as  WALLS,  DOORS,  FIXX1R,  and 
BASEBOARD,  have  well-defined  mutual  adjacencies  whereas  moveable  objects, 
such  as  WASTEBASKET  and  CHAIR,  can  appear  in  fairly  arbitrary  relationships 
with  each  other  and  with  the  fixtures.  The  analysis  also  used  the  constraint 
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DOOR  (DOOR  WALL  FLOOR  WASTEBASKET  TABLETOP)! 
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WASTEBASKET  (FLOOR  WALL  BASEBOARD  DOOR  TABLETOP  CHAIRBACK)) 
CHAIRBACK  (CHAIRSEAT  TABLETOP  WALL  DOOR  WASTEBASKET)) 


FIGURE  E-1  ADJACENCY  CONSTRAINTS  FOR  VERTICAL 
SURFACES  OF  ROOM  SCENES 
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ROOMPARTITION  to  eliminate  DOOR  and  WALL  as  possible  interpretations  of 
vertical  regions  with  limited  vertical  extent. 

Since  Waltz  dealt  strictly  with  symbolic  input,  the  classification 
routine  was  modified  to  return  an  a priori  likelihood  of  1.0  for  all  region 
interpretations  that  qualified  as  possibilities  based  on  their  height  and 
surface  orientation.  This  was  the  only  actual  modification  to  an  MSYS 
routine  required  to  emulate  the  Waltz  filtering  algorithm. 

Figure  E-2  contains  a complete  trace  of  the  interpretation  process 
encompassing  network  initialization  (lines  1 to  27)  and  evaluation  of  the 
resulting  solution  (lines  28  to  35).  A unique  and  consistent  interpretat ion 
of  the  scene  has  been  achieved  without  any  instantiation  and  with  consider- 
ably less  propagation  in  the  initialization  phase  than  was  required  to 
achieve  equilibrium  likelihoods  with  nondeterministie  constraints  (see 
Appendix  B).  The  final  equilibrium  likelihoods  for  all  interpretation 
variables  appear  in  Figure  E-3 , which  presents  the  data  base  at  equilibrium 
following  initialization. 

Figure  E-3  also  shows  the  procedures  that  were  compiled  from  the 
adjacency  constraints  of  Figure  E-l  for  computing  interpretation  likeli- 
hoods. These  procedures  contain  support  clauses  that  reduce  the  likeli- 
hood of  an  interpretation  to  zero  whenever  the  likelihoods  of  all  compatible 
interpretations  in  any  adjacent  region  become  zero.  The  evaluation  function 
for  (WALL  LWALL),  for  example,  contains  clauses  requiring  that  regions  DR 
be  DOOR,  that  region  PIC  be  PICTURE  or  DOOR,  that  region  WBSKT  be  DOOR  or 
WASTEBASKET,  and  that  region  TTOP  be  TABLETOP.  WALL  was  not  an  allowed 
interpretation  for  regions  DR  and  PIC  because  their  brightnesses  were 
markedly  different  from  that  of  region  LWALL. 
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OOOUl  BG-(NPTStTUP  (QUOTE  (WASTEBASKET  WESKT  ) ) ) : 1.0 

0000?  BC-CNETSETUP  (QUOTE  (OOOP  WBSKT ) ) ) : O.O-SETtJF 

00003  BG- ( IWE  TSE  TUP  (QUOJt  (aALL  WBSKT))):  0.0-SETUP 

00004  BG-CNETSFTUP  (QUOTE  (DOOH  P*ALL))):  1.0 

00005  RC,-(NETSF  TUP  (QUOTE  (WALL  RWALL ))) : 1.0 

0000b  BG- ( NETSF  TUP  ( QUOTE  (DOOR  LWALL ))  ) I 1.0 

00007  PG-fNETSETUP  (QUOTE  (WALL  LWALL))):  1.0 

00008  BG-(NETSETUP  (QUOTE  (DOOR  CbACK  ) ) ) : 0.0-SETUP 

00009  JOB  - ( ( (i  R * (aALL  CbACK)  (CHAIPBACK  CBACK)  (DOOR  CBACK))):  1.0 

00010  P<7-{  DiETSETUP  (QU01E  (CHAIPBACK  CBACK))):  ).o 

0001  1 BC»-(KK  TSETUP  (QUOTE  (WALL  CBACK))):  O.Q-SETUP 

0001  ? JOB  - ( ( OB*  (aALL  CRACK)  (CHAIPBACK  CBACK)  (DOOR  CBACK))):  1,0 

00013  PG-( NETS* TUP  (QUOTE  (PICTURE  PIC))):  1.0 

00014  BG» ( E E TSE  TOP  (QUOTE  (DOOR  PIC))):  0,0-SET UP 

00015  JOB  - ( (0R»  (DOOR  PIC)  (PICTURE  PIC))):  1.0 

0001b  6G- ( MET SET  UP  (QUOTE  (WALL  PIC))):  0. O-SETuP 

0001  7 JJE-((ANP*  (CP*  (WALL  wbSK'T)  (WASTEBASKET  W8SKT ) ) (TABLETOP  TTOP) 

00018  ( a A L f DP)  (WALL  PIC))):  0.0 

00019  JOB- ((POOP  Lw ALL  ) ) : 0.0 

0 0 0 20  J 0 B • ( (OP*  (aALL  LWALL)  (DOOR  LWALL))):  1.0 

00021  JOB-((AND*  (TABLETOP  TTOP)  (WALL  PIC))):  0.0 

00022  JOB-KDOUP  PWALI.)):  0.0 

00023  BG* (NET  SETUP  (QUOTE  (DOOR  DP))):  1.0 

00024  B G- ( NETSE 1 UP  (QUOTE  (aALL  DP))):  0.0 

00025  JOB-((ANC*  (OP*  (aAIL  WBSkT)  (WASTEBASKET  WBSKT))  (TABLETOP  TTOP) 
0002b  (WALL  DP)  (WALL  PIC))):  0.0 

00027  JOB-((OR*  (WALL  DP)  (DOOR  DR))):  1.0 

00028  BG- ( ( OPT  I Ob  (DOOR  DR)  (WALL  DP))):  (DOOR  DPJ-1.0 

00029  BG- ( ( OPT  1 ON  (PICTURE  PIC)  (DOOP  PIC)  (aALL  PIC))):  ( PICTURE  PTC)-t.O 

00030  hG- ( (OPT  I OK  (DOOP  CBACK)  (CHA1RBACK  CRACK)  (WALL  CRACK))):  (CHAIRBACK 

00031  CBACK)-t.O 

0003?  BG-(  (l)PTloK  (DOOP  LWALL)  (WALL  LWALL))):  (WALL  LWALLl-1,0 

00033  BG- ( (OPTION  (DOOP  RWALL)  (WALL  PaALL))!:  (WALL  PWALL)-1.0 

00034  FG-( (OPTION  (WASTEBASKET  wHSKT)  (DOOP  WBSKT)  (aALL  WBSKT))):  ( 

00035  WASTEBASKET  wBSKT)-1.0 

000  ib  "SUCCESS" 


FIGURE  E-2 
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VARIABLE:  (WASTEBASKET  WBSKT) 

VALUE:  1.0 

PROCFPURF; 


RELATIVES: 


TTCIP)  ) ) 


(AND*  1.0  (AND*  (TABLETOP  TTOP) 

(OR*  (WALL  C8ACK) 

(CHAIRBACK  CB*CK) 

(DOOR  CBACK)) 

(OR*  (WALL  DR) 

(DOOR  DP)) 

(OR*  (*ALL  LWALL) 

(DOOP  LwALL)) 

(FLOOR  FLR)  ) ) 

((OPTION  (WASTEBASKET  WBSKT ) (DOOP  WBSKT)  (WALL  WBSKT))) 
((OR*  (WALL  WBSKT)  (WASTEBASKET  WBSKT))) 

((UP*  (DOOR  WBSKT)  (WASTEBASKET  WBSKT))) 

((AND*  (CHA1PSEAT  CSEAT)  (WASTEBASKET  wBSKT ) (TABLETOP 

((OR*  (WALL  WBSKT)  (DOOR  WBSKT)  (WASTEBASKET  WBSKT))) 


VAPIABLEs  (DOOR  wBSKT) 

VALUE:  0.0 

PROCEDURE: 

(DOOR  WBSKT) 

RELATIVES: 

((OPTION  (WASTEBASKET  WBSKT)  (DOOP  WBSKT)  (WALL  WBSKT))) 
((OP*  (POOR  WBSKT)  (WASTEBASKET  wBSKT))) 

((OR*  (WALL  WBSKT)  (DOOR  WBSKT)  (WASTEBASKET  WBSKT))) 


VARIABLE:  (WALL  4BSKT) 

VALUE:  0.0 

procedure  : 

(WALL  WBSKT) 

RELATIVES: 

((OPTION  (WASTEBASKET  wBSKT)  (DOOR  WBSKT)  (WALL  WBSKT))) 
((OP*  (WALL  WRSKT)  (WASTEBASKET  WBSKT))) 

((OR*  (WALL  wBSKT)  (DOOR  WBSKT)  (wASTF. BASKET  WBSKT))) 


VARIABLE:  (DOOR  RWALL) 

VALUE:  0.0 

PROCEDURE: 

(and*  i.o  (and*  (Tabletop  ttopi 

(WALL  PIC))) 


RELATIVES: 

((OPTION  (DOOR  RWALL)  (WALL  RWALL))) 


VARIABLE:  (WALL  PwALL ) 
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00096 

PROCEDURE: 
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((OR*  (WALL  CBACK)  (CHAIPBACK  CBACK)  (DOOR  CBACK))) 

00101 

00102 

FIGURE  E-3  DATABASE  AT  INITIAL  EQUILIBRIUM  (Continued) 
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VARIABLE: 
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VALUE  : 
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VARIABLE: 
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VARIABLE: 
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VALUE  : 
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RELATIVES: 
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AN  MSYS  EMULATION  OK  DUD A’S  ALGORITHM 


Duda  formulated  scene  interpretation  as  a tree  searching  problem. 
Pictorial  regions  were  each  represented  by  a node  of  the  tree  and  the 
branches  emanating  from  a node  corresponded  to  the  possible  interpretations 
for  that  region.  The  first  region  selected  for  labeling  was  designated  as 
the  "start  node."  A path  through  the  tree  from  the  start  node  to  a terminal 
node  represented  a unique  labeling  of  the  scene. 

Every  region  interpretation  had  a likelihood  based  on  the  attributes 
(e.g.,  color  and  size)  of  its  associated  region  and  every  node  had  a score 
representing  the  sum  of  interpretation  likelihoods  along  the  path  from  the 
start  node  to  that  node.  Legal  interpretations  for  adjacent  regions  were 
constrained  deterministically,  as  in  the  Waltz  analysis  (see  Figure  E-l). 

An  A*  search  was  used  to  find  the  highest  scoring  path  through  the  tree 
that  satisfied  these  constraints. 

The  A*  search  proceeded  at  each  stage  by  expanding  the  open  node  with 
the  greatest  score.  Initially,  only  nodes  emanating  from  the  start  node, 
representing  possible  interpretations  of  the  first  region,  are  open.  To 
expand  a node,  a region  was  selected  that  was  not  previously  considered 
on  the  path  to  that  node.  The  expansion  node  was  removed  from  the  list 
of  open  nodes  and  new  open  nodes  were  added  for  each  interpretation  of 
the  selected  region  that  was  not  incompatible  with  any  previously  assigned 
region  interpretation  on  the  path.  Node  expansion  was  repeated  until  a 
terminal  node  was  selected  for  further  expansion;  the  path  to  that  terminal 
node  represented  the  highest  scoring  legal  labeling  of  the  scene. 

Duda's  algorithm  differs  from  M*  in  two  significant  ways.  First,  the 
selection  of  open  nodes  was  based  solely  on  the  a priori  likelihoods  of 
previously  instantiated  interpretations,  rather  than  on  globally  refined 
a posteriori  estimates  of  all  interpretation  likelihoods,  as  in  M*.  As 
in  the  Waltz  analysis,  constraints  were  strictly  Boolean  and  thus  could 
not  be  used  to  adjust  likelihoods,  except  to  zero.  Second,  Duda's  algorithm 
invoked  constraints  only  when  a node  was  expanded,  and  for  the  sole  purpose 
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of  determining  which  interpretations  of  the  selected  region  were  compatible 
with  the  other  interpretations  on  the  path  to  that  node.  Waltz,  and  M*,  by 
contrast,  do  not  require  instantiation  as  a prerequisite  for  invoking  a 
constraint.  Region  interpretations  are  eliminated  whenever  they  are  incon- 
sistent with  all  possible  interpretations  of  any  other  region.  This  policy 
allows  all  subtrees  containing  the  eliminated  interpretation  to  be  pruned 
from  the  search  and  avoids  redundant  discovery  of  the  same  inconsistency 
on  distinct  branches  of  the  search.  Moreover,  eliminations  can  propagate, 
allowing  additional  inconsistencies  to  be  discovered. 

Figures  F-l  through  F-4  document  the  interpretation  by  Duda's  algorithm 
of  the  scene  partition  depicted  in  Figure  4,  using  the  constraints  given  in 
Figure  E-l  (the  same  problem  previously'  analyzed  by  Waltz's  algorithm  in 
Appendix  E).  Figure  F-l  documents  the  initialization  phase.  Unlike  M* 
(Appendix  B)  and  Waltz  (Appendix  E),  no  likelihood  reevaluation  occurs 
because  constraints  do  not  apply  until  interpretations  are  instantiated. 
Figure  F-2,  a snapshot  of  the  data  base  following  initialization,  shows 
all  interpretations  still  carrying  their  a priori  likelihoods. 

In  the  search  phase  (Figure  F-3 ) , instantiations  are  proposed  on  the 
basis  of  the  a priori  interpretation  likelihoods.  As  a consequence,  the 
region  interpretation  (WALL  DR)  was  chosen  as  the  third  instantiation 
(Figure  F-3>  line  78)  and  the  search  was  forced  to  backtrack.  Note  that 
reevaluations  are  propagated  only  when  interpretations  are  asserted  and 
not  when  they  are  denied  because  only  in  the  case  of  a unique  instantiation 
are  new  constraints  activated.  In  the  terminal  state  (Figure  F-4),  the 
likelihoods  of  correct  interpretations  remain  at  their  a priori  values  while 
the  likelihoods  of  ail  other  interpretations  have  been  reduced  to  zero. 

The  emulation  of  Duda's  algorithm  in  MSYS  was  accomplished  by  modifying 
the  procedures  associated  with  region  interpretation  variables  and  OPTION 
variables.  The  OPTION  procedure  was  modified  so  that  only  the  likelihoods 
of  instantiated  interpretation  variables  contributed  to  the  score  of  a 
search  state.  The  modified  OPTION  procedure  returned  a Boolean  value,  1.0, 
if  the  associated  region  was  uninstantiated,  and  0.0,  if  it  was  instantiated. 
The  procedures  that  computed  interpretation  likelihoods  were  then  modified 
to  always  return  the  a priori  likelihood  of  the  interpretation  or  else  zero. 
The  likelihood  of  an  interpretation  was  'ero  only  if  that  interpretation  had 
been  instantiated,  and  an  adjacent  region  had  previously  been  instantiated 
to  an  incompatible  interpretation. 
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A comparison  of  the  likelihood  procedures  in  Figures  E-3  and  F-2 
shows  how  the  standard  M*  likelihood  procedures  used  in  the  Waltz  emula- 
tion were  modified  to  suppress  propagation  in  the  Duda  emulation.  Consider, 
in  particular,  the  procedures  associated  with  the  interpretation  variable 
(PICTURE  PIC)  at  line  125  in  Figure  E-3  and  line  188  in  Figure  F-2.  First, 
each  supporting  interpretation  of  a constraint  was  replaced  by  a disjunction 
(OR**)  of  that  interpretation  and  its  associated  OPTION  variable.  All  sup- 
port clauses  were  then  enclosed  in  a grand  disjunction  with  the  OPTION 
variable  of  the  constrained  interpretation  (PICTURE  PIC).  AND**  and  OR** 
were  threshold  versions  of  AND*  and  OR*,  respectively,  that  evaluated  to 
zero  or  one  depending  on  whether  or  not  the  corresponding  unthresholded 
function  evaluated  to  less  than  0.1.  Since  OPTION  variables  of  uninstantiated 
regions  have  the  value  1.0,  the  likelihood  of  (PICTURE  PIC)  is  pinned  at  the 
a priori  likelihood  0.12,  as  long  as  region  PIC  is  uninstantiated,  or  all 
of  the  following  conditions  apply:  Region  TTOP  is  uninstantiated  or 

instantiated  to  TABLETOP,  region  LWALL  is  uninstantiated  or  instantiated 
to  WALL,  and  region  RWALL  is  uninstantiated  or  instantiated  to  WALL. 

Otherwise,  the  likelihood  of  (PICTURE  PIC)  drops  to  zero. 
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i* 


0001'  1 
O'5no2 
00)03 
00004 
0 000=. 
OOO'io 
00007 
000  '*■ 

00000 
oooio 

0001  1 
000  1 2 
0001  3 
0001  4 
00015 

000  1 fe 

0001  7 
000  1 b 
0 0 01  0 
00020 
00021 
0 00  ? 2 

0002  3 
00024 
00  0 25 
00026 
0002  7 
00026 
00020 

00030 

00031 

00032 
000  33 
000  34 

000  36 
000  36 
000  37 
000  3 P 
0 003^ 
0 00  40 
000  4 ! 
0004? 

00043 

00044 

00045 

00046 

00047 
0 0 0 4 H 
00049 
00  05  0 
00051 


VARIABIE*  (WASTEBASKET  WBSKT) 

VALUE:  .12 

PROCEDURE  : 

[AND#  .12  (0P»*  (OPTION  (WASTEBASKET  WBSKT) 

( 0 0 0 R wBSKT) 

( WALL  WiBSK  T ) ) 

(AND**  (OR**  (OPTION  (TABLETOP  TTOP)) 
(TABLETOP  TTOP)) 

(OP**  (OPTION  (DOOP  CbACK) 

(CHAIPBACK  CBACK) 

(WALL  CBACK)) 

(WALL  CBACK) 

(CHAIPBACK  CBACK) 

(DOOR  CBACK)) 

(OR**  (OPTION  (DOOP  DR) 

(WALL  DR)) 

(wALL  DR) 

(DOOP  DR)) 

(0P»*  (OPTION  (DOOP  LWALL ) 

(WALL  LWALL)) 

(WALL  LWALL  ) 

(DOOP  LWALL)) 

(0P»»  (OPTION  (FLOOR  FLP)) 

(FLUOR  FLP  J 

PI LATIVES! 

( ( UPT 1 ON  (WASTEBASKET  WBSKT)  (DOOP  WBSKT)  (WALL  WBSKT ) ) ) 
((OR**  (OPTION  (WASTEBASKET  WBsKT)  (DOOP  wBSKT ) (WALL  WBSKT)) 
(WALL  wBSKT)  (WASTEBASKET  WBSKT))) 

((UP**  ( OP 1 ION  (WASTEBASKET  WBSKT)  (DOOP  WBSKT)  (WALL  wBSKT)) 
(0(309  wBSKT)  (wASTEHASKFT  WBSKT))) 

((UP**  (OPTION  (WASTEBASKET  WBsKT)  (DOOP  WBSKT)  (WALL  WBSKT)) 
(WASTEBASKET  WBSKT))) 

( ( (ip • • (OPTION  (WASTEBASKET  WBsKT)  (DOOP  WBSKT)  (WALL  WBSKT)) 
(WALL  WBSKT)  (DOOR  WBSKT)  (wASTEBASKET  WBsKT))) 


VARIABLE:  (DOOP  WBSKT) 

VALUE:  0.0 

FROCFIUIPF  : 

(DOOP  wBSKT) 

RELATIVES! 

((OPTION  (WASTEBASKET  WBSKT)  (DOOR  WBSKT)  (WALL  wBSKT ) ) ) 
({OP**  (OPTION  ( wASTEBASKET  wBSKT)  (OOOP  WBSKT)  (WALL  WBSKT)) 
(DOOP  WBSKT)  (WASTEBASKET  WBSKT))) 

(((JR**  (OPTION  (WASTEBASKET  WBsKT)  (DOOR  WBSKT)  (WALL  WBSKT)) 
(WAIT  WBSKT)  (DOOR  WBSKT)  (WASTEBASKET  WBSKT))) 


VARIABLE:  ( wALL  wBSKT) 

VALUE:  H.O 

PROCEDURE  : 
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0 0 Ob  2 
00(163 
0006  4 
OOOSS 
00066 
00067 
0006R 
0006R 
OOObO 
OOOe  I 
00062 

0006  3 
00064 
80065 
OOObb 

00067 

00068 
0 0 0 d 0 
00O7O 

0007  1 
0 0 0 7 2 
000  7 3 
00074 
000  7 5 

00076 

00077 
0 0 0 7 8 
000  7 0 
00080 

0008  I 
00082 
0008  3 
000  8 4 

00085 

00086 

00087 

00088 
00088 
00080 
00081 
00082 

00083 

00084 
OOO  36 
00086 

000  8 7 
00088 
00088 

001  00 
00101 
00102 


(RAIL  WBSKT) 

RELATIVE  S; 

((OPTION  ( W'ASTFBASKET  WBSKT)  (POOP  WBSKT)  (wALL  wBSKT))) 
((OR**  (OPTION  (WASTEBASKET  W&sKT)  (DOOR  wBSKT)  (WAIL  WBSKT  7 ) 
(WALl  WBSKT)  (WASTEBASKET  WBSKT))) 

( ( 0 P * * (OPTION  (WASTEBASKET  «BSKT ) (POOP  WBSKT)  (WALL  WBSKT)) 
(hALL  »BSKT)  (POOR  «BSKT)  ( WASTEBASKET  *BSK  T ) ) ) 


VARIABLE;  (POOR  RWALL) 

VALUF:  .227 

FROCFDURF • 

l A N L>  • . 227  (Op**  (OPTION  (POOP  RWALL) 

( WALL  RwALL  ) ) 

(AND**  ( OR**  (OPTION  (TABLETOP  TTOP)) 
(TABLETOP  TTOP)) 

(OR**  (OPTION  (PICTURE  PIC) 
(DOOP  PIC) 

(WALL  PIC)) 

(WALL  PIC1 

8 El  ATI VESs 

((OPTION  (DOOR  RwALL)  (WALL  R*ALL))) 


VAPIABI  E J (WALL  RWALL) 

VALUE : .773 

PROCEDURE ; 

[AND*  .773  (OR**  (OPTION  (DOOR  RWALL) 

(WALL  RWALL  ) ) 

(AND**  (OR**  (OPTION  (TABLE  TOP  TTOP)) 
(TABLETOP  TTOP)) 

(OR**  (OPTION  (PICTURE  PIC) 

(DOOR  PIC) 

(WALL  PIC)) 

(DOOR  PIC) 

(PICTURE  PIC) 

RELATIVES: 

((OPTION  (DOOR  RwALL)  (WALL  RwALL))) 

( (OP**  (OPTION  (DOOR  RWALL)  (WAU  RWALL))  (WALL  RWALL))) 


VARIABLE;  (DOOP  LWALI) 

VALUE : .227 

PR OCR DU PS ; 

LAND*  .227  (OR**  (OPTION  (DOOR  LwALL ) 

(WALL  LWALL) ) 

(AND**  (OP**  (OPTION  (WASTEBASKET  WBSKT) 

(POOR  WBSKT) 

(WALL  wBSKt)) 

(wALL  WBSKT) 

(WASTEBASKET  wBSKT)) 
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00103 

(OR**  (OPTION  ( 1 ABLE  TOP  TTOP)) 

00104 

(TABLETOP  TTOP)) 

001  06 

(OR**  (OPTION  (DOOR  DP) 

00  1 U6 

(HALL  DR)) 

00107 

('MALL  DR)) 

00108 

(OP##  ( OPT  T ON  (PICTURE  PIC) 

00  1 OR 

(DOOR  PIC) 

00110 

(WALL  RIC)  ) 

001  1 1 

(WAIL  PIC] 

00112 

RELATIVES: 

00113 

( (OPTION  (DOOR 

LaALL) 

(WALL  LWALL))) 

00114 

( ( UP  « • (OPTION 

(DOOR 

LNALL)  (WALL  LWALL)  ) (WALL  LWALL) 

00118 

(DOUR  L3ALUD 

00116 

( (UR*  * (OPTION 

( DOOR 

LWALL)  (WALL  LWALL))  (DOOR  LWALL))) 

00117 

00}  1 R 

00119 

VARIABLE:  ( 1~  ALL  LWALL) 

00120 

VALUE : .773 

00121 

PROCEDURE: 

00122 

(AND.  . 7 7 3 (OH#*  (OPTION  (DOOR  L*ALL  3 

00123 

(WALL  LWALL)) 

00124 

(AND##  (0R»*  (OPTION  (WASTEBASKET  WBSKT) 

00128 

(DOOR  aBSKT) 

001  7 6 

(WALL  WBSKT)) 

00127 

(DOOR  WBSKT) 

00  1 7 R 

(WASTEBASKET  WBSKT)) 

00129 

(OR**  (OPTION  (TABLETOP  TTOP)) 

00130 

(TABLETOP  TTOP)) 

00131 

(OP*»  (OPTION  (DOOP  DP) 

00132 

(WALL  DP)) 

00133 

l DOOR  DP)) 

001  34 

(OR*#  (OPTION  (FICTURP  PIC) 

001  75 

(DOOP  PIC) 

001  3b 

(WALL  PIC)) 

001  3 7 

(DOOR  PIC) 

001  3 P 

(PICTUPE  PIC] 

001  39 

RELATIVES: 

001  40 

((OPTION  (DOOR 

LWALL ) 

(wALL  LWALL))) 

00141 

((OR**  (OPTION 

(DOOR 

LwALL)  (WALL  LWALL))  (WALL  LWALL) 

00142 

(DOOR  L WALL  ) ) ) 

00143 

((OR**  (OPTION 

(DOOR 

lwall)  (wall  lwalljj  (wall  lwall))) 

00144 

00145 

001  46 

VARIABLE:  ( DOOR  C«ACK) 

00147 

VALUE:  0.0 

00  1 4 R 

PROCEDURE : 

00149 

(DOOR  CBACK ) 

00150 

RELATIVES: 

0015  1 

((OPTION  (DOOR 

CBACK) 

(CHAIPBACK  CBACK)  (WALL  CBACK))) 

00152 

((OR*#  (OPTION 

( DOOR 

CBACK)  (CHAIPBACK  CBACK)  (WALL  CBACK)) 

00153 

( a A LL  CBACK)  (CHA1PBACK  CBACK)  (DOOR  CRACK))) 
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0 0 t 5 4 

00155 

00156 

VARIABLE: 

(CHAIPB4CK  CBACK) 

00157 

V A l UE: 

. 1 1 

0 0 1 5 8 

P ROC FPU PI  : 

00150 

[AMD*  .11  (OP**  (OPTION  ( DiinR  CBACK) 

00160 

(CHAIPBACK  CBACK) 

00161 

(WALL  CBACK  ) ) 

00162 

(AND**  (OR**  (OPTION  (CHAIRSEAT  CSEAT1) 

0016  3 

(CHAIRSEAT  C SEAT)  ) 

00164 

(OR**  (OPTION  (WASTEBASKET  WBSKT) 

00165 

(DOOR  WBSKT) 

00  166 

(RALL  WBSKT)) 

00167 

(wASTEBASKEI  WBSKT)) 

00168 

(OR**  (OPTION  (TABLETOP  TTOP)) 

00169 

(TABLETOP  TTOP) 

0 017  0 

P 1 1 a T I V E s : 

00171 

((OPTION  (DOOR  CBACK)  (CHAIPBACK  CBACK)  (WALL  CBACK))) 

00172 

((OR**  (OPTION  (DOOR  CBACK)  (CHAIPBACK  CBACK)  (WALL  CBACK)) 

0017? 

(WALL  CRACK  ) (CHA1RRACK  CBACK)  (DOOR  CBACK))) 

0 017  4 

00175 

00176 

VAF  TABLE  : 

( WALL  CBACK  ) 

00177 

V AI  OF  : 

O.U 

001  74 

FROCEDDRF  ; 

001  74 

(WALL  CBACK) 

001»0 

FF.LATIVES: 

00181 

((OPTION  (DOOR  CBACK)  (CHAIPBACK.  CBACK)  (WALL  CBACK))) 

00182 

((OR**  (OPTION  (DOOR  CRACK)  (CHAIPBACK  CbACK)  (WALL  CBACK)) 

00187 

(aAI.L  CBACK)  (CHAIPBACK  CRACK)  (DOOR  CBACKJ1) 

00 184 

00185 

00186 

VAFIARLF : 

( P I C Ul  P t PIC) 

00  167 

VALUE : 

. 3 

00188 

PROCEDURE ; 

00184 

(AMD*  .3  (OP**  (OPTION  (PICTURE  PIC) 

00190 

(DOOP  PIC) 

00141 

(WALL  PIC  )) 

00142 

(AND**  (OP**  (OPTION  (TABLETOP  TTOP)) 

00193 

(TABLETOP  TTOP)) 

00194 

(OR**  (OPTION  (DOOR  LWALL ) 

00145 

(WALL  LWALL)  ) 

00  1 96 

(WALL  LWALL)) 

00  197 

(OP**  (OPTION  (DOOF.  PW'ALL ) 

00198 

(WALL  P W ALL ) ) 

00  t 99 

(WALL  RWALLI 

0 0 20  0 

R F l A I I V F S : 

00201 

((OPTION  (PICTURE  PIC)  (DOOP  PIC)  (WALL  PIC))) 

00202 

( ( OP  * * (OPTION  (PICTURE  PIC)  (OOUR  PIC)  (WALL  PIC))  (DOOR 

00203 

PIC)  (PICTURE  PIC))) 

00204 

FIGURE  F-2  DATABASE  PRIOR  TO  SEARCH 
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002Ot 
00  70b 

VARIABLE: 

(POOP  PIC) 

00207 

VALUfc  : 

0.0 

00208 

PROCEDURE  : 

00209 

(DOOR  PIC) 

00210 

RELATIVES; 

0021  1 

((OPTION  (PICTURE  PIC)  (DOOR  PIC)  (WALL  PIC))) 

0021  2 

((OP**  ( OF 1 I ON  (PICTURE  PIC)  (DOOR  PIC)  (BALL  PIC)) 

0021  3 

PIC)  (PICTURE  PIC))) 

0021  4 
0021  5 
0021b 

VAFIABI E : 

(WALL  PIC) 

0021  7 

VALUE: 

0.0 

0021  8 

PROCEDURE : 

0071  a 

(WALL  PIC) 

00220 

RELATIVES: 

00271 

((OPTION  (PICTURE  PIC)  (DOOR  PIC)  (WALL  PIC))) 

00727 

((OR**  (OPTION  (FICTUPE  PIC)  (DOOR  PIC)  (WALL  PIC)) 

0 0 2 7 1 

PIC  ) ) ) 

00224 

00275 

0027b 

VARIABLE: 

(DOOR  DR) 

00727 

VALUE: 

.727 

00228 

PROCEDURE : 

00279 

f AND*  . 277  (OR**  (OPTION  (DOOR  OH) 

002  JO 

(WALL  DR)  ) 

00231 

(AND**  (OR**  (OPTION  (WASTEBASKET 

002  12 

(DOOR  WBSKT ) 

007  13 

(WALL  wBSKT)  ) 

002  14 

(WALL  WBSKT) 

002  15 

(DOOR  wBSKT ) 

002  lb 

(WASTEBASKET  WBSKT)) 

007  37 

(0R*«  (OPTION  (DOOR  LWALL) 

002  38 

(WALL  LWALL)) 

007  19 

(WALL  LWALL)) 

00240 

(OR**  (OPTION  (ELUOP  FLR ) ) 

00241 

(FLOUR  FLR] 

00242 

RELATIVES: 

0074  3 

((OPTION  (DOOR  OR)  (WALL  DR))) 

00244 

((OR**  (OPTION  (DOOR  DR)  (wALL  DR))  (WALL  OP)  (DOOR 

00745 

((UR**  (OPTION  (DOOR  DP)  (WALL  OH))  (DOOR  DR))) 

0024b 

00247 

00248 

VARIABLE; 

(WALL  DP) 

00249 

VALUE : 

.77  3 

00750 

PROCEDURE : 

00251 

1 A N D * . 77  3 (OR**  (OPTION  (DOOR  DR) 

00252 

(wALL  DR)  ) 

00253 

(AND**  (OP**  (OPTION  (WASTERASKFT 

00254 

(DOOR  WBSKT) 

00255 

(WALL  WBSKT)) 
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00256 

00257 

00258 
00759 
00260 
00261 
00262 

00263 

00264 

00265 

00266 
00267 


(WALL  WBSKT) 

(DOOR  WBSKT) 

(WASTEBASKET  WBSKT ) ) 

( 08**  (OPTION  (DOOP  LWALL) 

(WALL  LW ALL  5 ) 

( DOOP  LWALL  ) 1 

(UR**  (OPTION  ( FLOOR  FLP ) 3 
(FLOOR  FLP] 

Pf  LAT1VFS: 

((OPTION  (DOOR  DPI  ( a ALL  DPI)) 

((OR**  (OPTION  (LOOP  OR)  (aALL  DR))  (WALL  DR)  (DOOR  DP))) 
l ( OR  * * (OFTION  (DOOR  DR)  (WALL  DR))  (WALL  DR))) 
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00030 
000  3 1 

00032 

00033 

00034 
000  3b 
000  3b 
00O  37 
00038 

000  34 

00040 

00041 

00042 

00043 

00044 
0 0 0 4b 
0004b 

00047 

00048 

00049 

00050 
000b  1 

00052 

00053 

00054 

00055 
0005b 
000b  7 

00058 

00059 
0 0 0 b 0 
000b  1 

00052 
000b  3 
000b4 
00055 
OOObh 
000b  7 
00068 

00069 

00070 
0007  1 

00072 

00073 

00074 

00075 
0007b 

00077 

00078 

00079 

00080 


88  INSTATE  : (TOPCNTXT) 

ASSERT  - (WAIL  RWALL) 

hG- ( (OPTION  (DOUR  8 WALL ) (WALL  8WALL  ) I ) : 0.0 

JUP-((08«»  (OPTION  (DOOR  RWALL)  (WALL  RwALL))  (WALL  R « ALL  JO):  1.0 

JOB- ( (OR**  (OPTION  (DOOR  RWALL)  (WALL  RWAIL))  ( AND**  (OP**  (OPTION 

( TABLET  OP  TTOP))  (TABLETOP  TTOP))  (OR**  (OPTION  (PICTURE  PIC)  (DOOR 

PIC)  (WALL  PIC))  (DOOR  PIC)  (PICTURE  PIC)))));  1.0 

5AV ESTATE : (((WALL  RWALL)  . .773)  TOPCNTXT),  SCOPE:  .773 

, LENGTH(  IOUEUt)  : 5 

REINSTATE : ( TOPCNTXT ) 

DENY  - (WALL.  PWALI  ) 

JOB  - ( (OR**  (OPTION  (DOOP  PWALL ) (WALL  PHALLI)  (WALL  RWALL))):  1.0 
BG*  ( (OPTION  (DOOP  PWALL ) (WALL  RWALL))):  1.0 
SAVESTATE:  (((WALL  PWALL)  . 0)  TOPCNTXT),  SCORE;  0, 

REINSTATE.;  (((«ALL  RwAIL)  . . 77  3 ) TOPCNTXT) 

ASSERT  - (WALL  L w ALL ) 

JOB* ( (OR**  (OPTION  (DOOP  LWALL)  (WALL  L WA  LL ) ) (DOOP 
J(iP-((OP«»  (OPTION  (DOOR  LwALL)  (WALL  LWALL))  (WALI 


LENGTH ( IOUEUE) : 6 


LWALL))):  1.0 
LWALL)  (DOOR  LWALL))) 


: 1.0 

RG*( (OPTION  (DOOP  LwALL)  (WALL  LWALL))):  0.0 

JOB-((OR»*  (OPTION  (DOOR  LWALL)  (WAIL  L*  ALL ) ) (DOOR  LWALL))):  0,0 
jnb»((AND««  (OR**  (OPTION  ( WASTEBASKET  *BSKT)  (DOOP  WBSKT)  (WALL  WBSKT ) ) 
(WALL  WBSKT ) (DOOR  WBSKT)  (wASTEBASKET  WBSKT))  (GP«*  (OPTION  (DOOR 
LwALl)  (WAIL  LWALL))  (DOOP  LWALL))  (OP**  (OPTION  (FLOOR  ELP ) ) (FLOOR 
F LP  ) ) ) ) : 0.0 

JOB • ( ( 08*  * (OPTION  (DOOR  08 ) (WALL  DP))  (AND**  (OP**  (OPTION  ( 

WASTEBASKET  WbSKl)  (DOOP  WBSKT)  ( wALL  WBSKT))  (WALL  WBSKT)  (DOOR  WBSKT) 
(WASTEBASKET  wrtSKl)}  (OF**  (OPTION  (DOOP  LWALL)  (WALL  LWALL) ) (DOOP 
LwALl))  (OF**  (OPTION  (FLOOp  FIR))  (FLOOR  FLP})))):  1.0 
J0B-((C18**  (OPTION  (DOOR  LWALL)  (wALL  I WALI))  (WALL  LWALL))):  1.0 
jne-((OP«*  (OPTION  (DOOP  LWALL)  (WALL  I.wALL))  (AND**  (OP**  (OPTION 
( *ASTEBASKET  WBSKV)  (DOOP  WBSKT)  (WALI  WBgKT))  (TOOP  WBSKT)  (WASTEBASKET 
WBSKT))  (OP«*  (OPTION  (TABLETOP  TTOP))  (TABLETOP  TTOP))  (OP**  (OPTION 
(DOOR  DP)  (WALL  DR))  (DOOP  DR))  (OR**  (OPTION  ( PICTURE  PIC)  (DOOR 
PIC)  ( wAI I PIC))  (DOOR  PIC)  (PICTURE  PIC))))):  1.0 

IOB-((DP**  (OPTION  (DOOR  LwALL)  (WALL  L*A1,L))  (WALL  LWALL)  (DOOR  LWALL))) 


: 1 .0 

SAVESTATF:  (((WALL  LWALL)  . ,713)  ((WALL.  PWALL)  . .773)  TOPCNTXT) 

, SCOPE:  1.55,  LENGTH(IOUEUE)  : 4 

REINSTATE:  (((WALL  PWALL)  . .773)  TOPCNTXT) 

DENY  - (WALL  LWALL) 

JOB-((UF»*  (OPTION  (DOOP  LWALL)  (wALL  LwAl.L))  (WALI  LWALL))):  1.0 

vlOP-((OP»*  (OPTION  (DOOR  LWALL)  (WALL  LwALL))  (WALL  LWALL)  (DOOP  LWALL))) 

: 1 .0 

RG-KOHION  (DOOR  LWALL)  (WALL  LWALL))):  1.0 

S A VEST  ATE:  (((wALL  L*  ALL ) . 0)  ((WALL  RwALL)  . .773  ) TOPCNTXT),  SCORE:  .773 
. lENGTHf IOUEUE)  : 5 

REINSTATE:  (((WALL  LWALL)  . .773)  ((WALL  PWALL)  . .773)  TOPCNTXT) 

ASSERT  - (WALL  DP) 

JOB-((np**  (OPTION  (DOOP  DP)  (WALL  DR))  (DOOF  DP))):  1.0 

jriR-((OP*#  (OPTION  (DOOP  DP)  (wALL  DP))  (WALL  DP)  (DOOR  DP))):  1.0 
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00081  B G - ( (OPTION  (DOOP  DP)  (PALL  DR))):  0.0 

000®  2 JOB  • ( ( (iP  • * (OPTION  (DOOR  DP)  (PALI,  DP))  ( AND*»  (OP*«  (OPTION  ( 

000P)  WASTEHASKFT  WBSKT)  (DOOR  WBSKT)  ( w ALL  WbSKT))  (WALL  WBSKT)  (DOOR  wBSKT) 

00084  (PASTE BASK! T WBSKT ) ) (OR*»  (OPTION  (DOOR  LPALL ) (PALL  LWALL ) ) (DOOP 

00085  LPALL))  (OB**  (OPTION  (FLOOR  FLP11  (FLOOP  FLP ) ) ) ) ) ; 0.0 

0008b  J 0 B - ( (wALL  Dp)):  0.0 

00087  JOB  - ( f OR  • • (OPTION  (DOOR  DR)  ( w ALL  DR))  (PALI  DP)  (DOOR  DP))):  0.0 

00088  JOB- ( ( AND**  (OR**  (OPTION  ( TABLFTOP  TTOP))  (TAPLFTOP  TTOP))  (OP** 

00 0 8 R (OPTION  (DOOP  CBACK)  (CHAIPBACK  CBACK ) (WALL  CRACK))  (WALL  CBACK) 

00080  (CHAIPBACK  CBACK)  (DOOR  CBACK))  (OP**  (OPTION  (DOOF  DR)  (WALL  DP)) 

00081  (WALL  DR)  (DOOR  DR))  (OR**  (OPTION  (DOijp  LWALL)  (WALL  LWALL))  (WALL 

00082  LWALL)  (DOOP  LwALL) ) (OP**  (OPTION  (FLOOR  FLP))  (FLOOR  FLH))1):  0.0 

00083  J)B-( (OP**  (OPTION  (WASTEBASKET  WBSKT)  (DOOR  WBSKT)  (WALL  WBSKT ) ) 

00084  (AND**  (OR**  (OPTION  (TABLETOP  T TOF ) ) (TABLETOP  TTOP))  (OR**  (OPTION 

00085  (LOOP  CBACK)  (CHAIPBACK  CBACK)  (WALL  CBACK))  (WALL  CBACK)  (CHAIPBACK 

00096  CBACK)  (DOOP  CBACK))  (OP*»  (OPTION  (DOilP  OP)  ( * ALL  DP))  (WALL  DP) 

00097  (DOOP  DR))  (OP**  (OPTION  (OOOR  LWALL)  ( -ALL  LWALL))  (WALI  LwALL)  (DOOP 

0009P  l WALL))  (OP*«  (OPHON  (FLOOP  F L R ) ) (FLOOP  FLP))))):  1.0 

00099  JGB-((OP**  (OPTION  (DOOR  DP)  ( w A LL  DR)1  (DOOP  DP))):  0.0 

00100  JOB-1 (ANT**  (UP**  (OPTION  ( v ASTEBASKE I wBSKT)  (DOOP  WBSKT)  (WALL  WBSKT)) 

00101  (DOOP  wBSKT)  (WASTEBASKET  WbSK  T ) ) (OP««  (OPTION  (TABLETOP  TTOP))  ( 

00102  TABLFTOP  TTOP))  (OP**  (OPTION  (DOOP  DP)  (*ALt  t'P))  (DOOP  TP))  (OR** 

00103  (OPTION  (PICTURE  PIC)  (DOOP  PIC)  (WALL  PIC))  (DOOR  PIC)  (PICTUPF  PIC)))) 

00104  : 0.0 

001  05  JOB-(((iR*»  (OPTION  (DOOR  LWALL)  (WALL  L-'AlL))  (AM)**  (OP**  (OPTION 

00106  (WASTEBASKET  wBSKT)  (DOOP  wBSKT)  (WALL  WBSKT))  (DOOR  WBSKT)  (WASTEBASKET 

00107  WBSKT))  (OP**  (OPTION  (TABLETOP  TTOP))  (TAbLFTOP  TTOP))  (OP**  (OPTION 
001OR  (LOOP  DR)  (WALL  DR))  (DOPp  DP))  (OR**  (OPTION  (PICTURE  PIC)  (DOOP 

00109  PIC)  (WALL  TIC))  (DOOR  PTC)  (PICTURE  FTC))))):  0.0 

00110  J 0 B - ( (WALL  LWALL)):  O.C 

00111  JOB-KOR**  ( OFTI ON  (DOOR  LWAI.I,)  (wALL  LWALL))  (WALL  LwALL))).  o.O 

00112  JOB-( (AND**  (OP**  (OPTION  ( I ABL.ETOP  TTOP))  (TABLETOP  TTOP))  (OR** 

00113  (OPTION  (DOOF  LwALL)  (WALL  LWALL))  (wALL  LWAIL))  (OR**  (OPTION  (DOOR 

00114  PwALt  ) (WALL  P w A LL ) ) (WALL  RWAI.L)))):  0.0 

00115  JOB-((OR**  (OPTION  (PICTURE  PIC)  (Dih)K  PIC)  (WALL  PTC))  (AND**  (OR** 

00116  (OPT10F  (TABLFTOP  TTLP1)  (TABLE TOP  TTOP))  (OR**  (OPTlnN  (DOOR  LWALL) 

00117  ( w A I L.  1WALL))  (wALl  LwALL)  ) (3P»*  (OPTION  (DOOR  PwALL)  (WALL  P W A L L ) ) 

001  1 P (WALL  P W A I I, ) ) ) ) ) : 1.0 

00119  jnb - ( (OR**  (OPTION  (THOR  LWALL)  (wALL  LwALL))  (WALL  LwALL)  (DOOR  LWALL))) 

00120  ; n.p 

00121  JOB-((AND»*  (OF**  (OPHON  (TABLFTOP  TTOP))  (TABLETOP  TIOP)l  (OP** 

00122  (OPTION  (POUF  CBACK)  (CHAIPBACK  CBACK)  (WAIT  CBACK))  (WALL  CBACK) 

00123  (CHAIPBACK  CPACF)  (DOnp  CHACK))  (OP**  (OPTIOf  (DOOP  DR)  (WALL  DP)) 

001  24  ( w A LI  DR)  (LOUR  DP))  (OP.*  (OPTION  (DOOP  LWAIL)  (WALL  LwALL))  (WALL 

001  25  IwALL)  (POOP  LwALL))  (OP*.  (OPTION  (FLOOR  FLF ) ) (FLOOR  FLP)))):  0.0 

00126  BG- ( ( OPT  I ON  (DOOR  DR)  (WALL  DR))):  "CONTRADICTION  DETEC  I FD • 

001  27  PFIN STATE : (((WALL  LWALL)  . .77))  ((WALL  PWALL ) . . 77  3 ) TOFCNTXT) 


DENY  - (WALL  DP) 

JPB-( (OP*»  (OPTION  (DOOR  DPI  (wALl,  DR))  (wALL  DP)  (DOOR  DP))):  1.0 
BG-( (OPTION  (DOOP  DP)  (WALL  DP)));  1,0 

SAVESTATE:  (((WALL  DP)  . 0)  ((WALL  LWAIL)  . .7T3)  ((WALL  PwALL)  . 
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. 77  3 ) TOPCNTXT),  SCOPE:  1.55,  LENGTH { I QUEUE ) : 4 

REINSTATE;  (((WALL  DP)  . 0)  ((WALL  LWALL ) . .773)  ((WALL  RWALL)  . 

.77})  TOPCNTXT) 

ASSERT  - (F1CTUPE  PIC) 

S&VESTATE:  ( ( (PICTURE  PIC)  . .3)  ((wALL  DR)  . 0 ) ((WALL  LWALL)  . .773) 
((WALL  PWALL)  . . 77  3 ) TOPCNTXT),  SCORE;  1.55,  LENGTH ( I QUEUE ) : 3 
REINSTATE:  (((WALL  DP)  . n)  ((WALL  LWALL)  . .773)  ((WALL  RWALL)  . 

.773)  TOPCNTXT) 

DENY  * (PICTUPE  PIC) 

JOB  - ( (OR#*  (OPTION  (PICTURF  PIC)  (DOOR  PIC)  (WALL  PIC))  (DOOR  PIC) 
(PICTURE  PIC))):  1.0 

RG-(((lpriON  (PICTURE  PIC)  (DOOR  PIC)  (wALL  PIC))): 

"CONTRADICTION  DETECTED" 

PF.  INSTATE : (((PICTURE  PIC)  . .3)  ((wALL  DR)  . 0)  ((WALL  LwALL)  . ,77  3 ) 

((wALL  PWALL)  . .773)  TOPCNrXT) 

ASSERT  - (DOOR  DR) 

SAVESTATE:  (((DOOR  OP)  . .227)  ((PICTURE  PIC)  . .3)  ((WALL  DR)  . 0) 
((WAIL  LWALl ) . .773)  ((WALL  PWALL)  . .773)  TOPCNTXT),  SCOPE:  1.55 
, LENGTH ( IGUEUE) : 2 

REINSTATE:  (((PICTURE  PIC)  . .3)  ((wALL  OR)  . 0)  ((WALL  LwALL)  . .773) 

((WALL  RwALI  ) . , 77  3 ) TOPCNTXT) 

DENY  - (DOOR  OP) 

JOB-((OR«*  (OPTION  (DOOP  DR)  (wALL  DP))  (DOOR  DR))):  1.0 
JOB-((OP*»  (OPTION  (DOOP  DP)  (WALL  DR))  (WALL  DP)  (DOOP  DR))):  1.0 
BG* ( ( OPT 1 ON  (DOOR  DR)  (WALL  DR))):  "CONTRADICTION  DETECTED" 

REINSTATE:  (((DOOR  DP)  . .227)  ((PICTURE  PIC)  . .3)  ((WALL  DP)  . 0) 

( (WAIL  (.WALL)  . . 7 7 3 ) ((WALL  RWALL)  . .77})  TOPCNTXT) 

ASSERT  - (WASTEBASKET  WBSKT) 

SAVESTATE:  ( ( ( W ASTFB ASKET  wRSKT 1 . .12)  ((DOOR  UP)  . .227)  ( (PICTURE 
PIC)  . .3)  ((wALL  DR)  . 0)  ((WAIL  LWALL)  . . 7 7 3 ) ((wALL  PWALL)  . .7  7 3 ) 
1UFCNTXT),  SCORE:  1.55,  LENGTH ( I QUEUE  ) : 1 

REINSTATE:  (((DOOR  DR)  . .227)  ( (PICTURE  PIC)  . .3)  ( (WALL  DR)  , 0) 

({WALL  LWALL)  . .773)  ((WALL  RWALL)  . .77})  TOPCNTXT) 

DENY  - (WASTEBASKET  WBSKT) 

JOP-((OP*»  (OPTION  (WASTEBASKET  WBSKT)  (DOOR  VBSKT)  (WALL  WBSnT)) 

(WALI  * HSK  f ) (DOOR  wBSKT)  (WASTEBASKET  wPsKT))):  1.0 

,)ne-((OP«»  (OPTION  (WASTEBASKET  WBSKT)  (DOOP  WBSKT)  (WALL  WBSKT)) 

(WASTEBASKET  WBSKT)));  1.0 

JOB-((OP**  (OPTION  (WASTEBASKET  wBSKT ) (DOOR  WBSKT)  (WALL  WBSKT)) 

(HOUR  WBSKT)  (WASTEBASKET  WBSKT))):  1.0 

PC” ( (OPTION  (wASTtBASKET  wBSKT)  (DOOP  wBSKT)  (WALL  WBSKT))): 
"CONTRADICTION  DETECTED" 

PfINSTATF.:  ( ( (wASIEBASKET  WBSKT)  . .12)  ((DOOP  DP)  . .227  ) ((PICTURE 
PIC)  . .3)  ((WAIL  DR)  . 0)  ((wALL  LWALL)  . .773)  ((WALL  PwALL)  . .773) 
TOPCNTXT) 

ASSERT  - (CHAIPBACK  CoACK) 

"SUCCESS" 
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00001 

00002 

00003 

00004 

oooos 

00006 

00007 

OOOOP 

00009 

00010 
0001  1 
00012 
0001  3 
0001  4 
0001% 
00016 
00017 
0001  H 

00019 

00020 
00021 
00022 

00023 

00024 

00025 

00026 
00027 
000  2 R 
00029 
000  30 
00031 
000  32 
000  33 
00034 
00036 
00036 
000  37 
000  3 R 

00039 

00040 

00041 

00042 

00043 

00044 

00045 

00046 
0004  7 
0004P 
00049 
00060 
00061 


VARIABLE:  (WASTEBASKET  WBSKTl 

VALUE:  .12 

procedure: 

[AND#  .12  COR**  (OPTION  (WASTEBASKET  WBSKTl 

(DilOH  WBSKT) 

(WALL  WBSKT)) 

(AND**  (OP**  (OPTION  (TABLETOP  TT0P1) 
(TABLETOP  TTOP ) ) 

(OP**  (OPTION  (DOOR  CBACK) 

(CHAIRHACK  CBACK) 

(WALL  CBACK)) 

(WALL  CBACK) 

(CHA1RBACK  CBACK) 

(DOOR  CBACK)) 

(OP**  (OPTION  (DOOR  DR) 

(WALL  DP)  ) 

( « ALL  PR) 

(DOOR  DP)) 

(OP**  (OPTION  (DOOR  LWALl.) 

( *ALL  L* ALL  ) ) 

( MALI  I WALL) 

(DOOR  IWALL)) 

(OR**  (OPTION  (FLOOR  FLR ) ) 

(FLOOR  FLR 1 

RELATIVES: 

( ( OF  T I ON  ( *ASTEBA.6KFT  wBSKT)  (DOOR  WriSKT)  (WALL  wBSKT))) 

((OP**  (ORTION  ( wASTFBASKET  WBsKT)  (DOOR  WBSKT)  (WALL  WBSKT ) ) 
(DOOR  WBSKT)  (wASURASKFT  WBSKT))) 

( (OR**  (OPTION  (WASTEBASKET  WBSKT)  (ODOR  W’BSKT)  (wALL  WBSKT)) 
(WAS]  EF’A.SK  E T WHsM))) 

((OP**  ( OFT  I ON  ( * A ST t B A SKFT  WBSKT)  (DOOR  WBSKT)  (WALL  WBSKT)) 
(WALL  WBSKT)  (DOOR  wBSKT  ) ( w ASTEB  A SK  F.T  wBSKT))) 


VARIABLE;  (DOOR  wBSKT) 

VALIIR:  0.0 

RRPCfc  DUPE  : 

(POOR  wBSKT) 

FELATIVFS: 

((OPTION  (wASTF.BASKET  WBSKT)  (DOOP  WBSKT)  (WALL  WBSKT))) 

( ( 0 R • • (OPTION  (WASTEBASKET  WBSKT)  (POOP  WBSKT)  (WALL  WBSKT)) 
(POOR  wHSKT)  ( wAS  rEPASKFT  WBSKT))) 

((OR**  (OPHON  (wASTEBASKFT  WBSKT)  (POOP  WBSKT)  (WALL  WBSKT)) 
(WALL  wBSKT)  (DOOP  wBSKT)  (WASTEBASKET  wBSKT))) 


VARIABLE:  (WALL  wBSKT  ) 

VALUE!  0.0 

PROCEDURE  : 

(WALL  WBSKT) 

RELATIVES: 
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00052 

((OPTION  (WASTEBASKET  WBSKT)  (DOOP  WBSKT)  (WALL  WBSKT  ))  ) 

00053 

( (OP**  (OPTION  ( *A  STEB ASKE  T WBSKT ) (DOOP  WBSKT)  (WALL  WBSKT)) 

00054 

00055 

00056 

( wALl  WBSKT)  (DOOP  WBSKT)  C WASTEBASKET  wBSKT))) 

00057 

V A P I ABLE: 

(DOOR  PWALL) 

00050 

VALUE: 

0.0 

0005<J 

procedure : 

000  oO 

l A N D • .227  (OR**  ( OP  I ION  (DOOR  PWALL) 

0006  1 

CwALL  PWALL)) 

00062 

(AND**  (OP**  (OPTION  (TABLETOP  TTOP)) 

00063 

(TABLETOP  TTOP)) 

00064 

(OP**  (OPTION  (PICTURE  PIC) 

00066 

(DOOR  PIC) 

00066 

(WALL  PTC)) 

00067 

(WATL  PIC) 

0006B 

PFLA7 IVES: 

00069 

00070 
0007  1 

((OPTION  (DOIJP  Pa  ALL ) (WALL  RwALL))) 

00072 

VAPIAPU  : 

(WALL  PwALL) 

00073 

VALUE : 

.773 

00074 

PROCEDURE : 

0007S 

r AND*  .773  (OP**  (OPTION  (DOOP  PWALL) 

00076 

(WALL  PwALL)) 

00077 

(AND**  (OP**  (OPTION  (TABLETOP  TTOP)) 

00076 

(TABLE  TOP  TTOP)) 

00079 

(OP**  (OPTION  (PICTURE  PTC) 

000  8 0 

(DOOP  PIC) 

0006  1 

(WALL  PIC)) 

00082 

(DOOP  PIC) 

O00H  3 

(PICTURE  PIC) 

00084 

PEL AT  I VES : 

00085 

((OPTION  (DOOR  PwALL)  (WALL  RWALL))) 

00086 
000  B 7 
00086 

((OP**  (OPTION  (DOOR  PWALL)  ( WALL  PWALL))  (WALL  PWALL))) 

00089 

VAP  TABLE: 

(DO UP  LaALL) 

00090 

VAL  Ilf  : 

0.0 

00091 

ppnCEtHipE: 

00O92 

[AND*  .227  (OP»*  (OPTION  (DOTH  LwALL ) 

00093 

(WALL  LWALL)) 

00094 

(AND**  (OP**  (OPTION  (WASTEBASKET  WBSKT) 

00096 

(DOOP  WBSKT) 

00096 

(WALL  WBSKT)) 

00097 

(WALT,  WBSKT) 

00098 

(WASTEBASKE1  WBSKT)) 

00099 

(OP**  (OPTION  (TABLETOP  ITOP)) 

00100 

(TABLETOP  TTOP)) 

00IU1 

( OP*  * (OPTION  (DOOP  DR) 

00102 

(WALL  DR)) 
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00103 

(WAIL  DP)) 

00104 

(OP**  (OPTION  (PICTURE  PIC) 

00105 

(DOOR  PIC) 

00 1 OB 

(WALL  PIC)) 

00107 

(WALL  PIC) 

00108 

RELATIVES; 

00109 

((OPTION  (DOOP 

LwALL)  (WALL  l* ALL ) ) ) 

001  10 

(fOP*»  (OPTION 

(DOOP  LWALL)  (WALL  LW ALL ) ) (WALL  LWALL) 

00111 

00112 

00113 

( HOOP  LWALL ) ) ) 

00114 

VARIABLE;  ( WALL  LWALL) 

00115 

V A 1 OF  ; .773 

00116 

PROCEDURE; 

00117 

l»ND*  .771  (OR**  ( OP  1 I ON  (OQOP  LWALL) 

00118 

(WALL  LWALL)) 

00119 

(AND**  (OR**  (OPTION  (WASTEBASKET  WBSKT ) 

00120 

(DOOP  WBSKT) 

00121 

(WALL  WBSKT)) 

00122 

(DOOP  WBSKT) 

00123 

(WASTEBASKET  WBSKT)) 

00124 

( 0D* * (OPTION  (TABLETOP  TTOP)) 

00125 

(TABLETOP  TTOP)) 

00126 

(OR**  (OPTION  (OOOP  OP) 

00127 

(WALL  DR)) 

00128 

(DOOP  DR)) 

00129 

(OR**  (OPTION  (PICTURE  PIC) 

00130 

(DOOR  PIC) 

00131 

(WALL  PIC)) 

001  12 

(Ooop  pro 

00133 

(PICTURE  PIC) 

001  14 

8 FI ATIVES; 

001  35 

((OPTION  (OOOP 

LWALL)  (wALL  LwALL))) 

001  36 

C ( 1 *P  *•  (OPTION 

(DOOR  LWALL)  (WALL  LwALL))  (WALL  LWALL) 

001  37 

( poop  LWALL )) ) 

00138 
001  19 
00110 

( (OP**  (OPTION 

(DOOR  LwALL)  (WALL  LWALL))  (WALL  LWALL))) 

00141 

VAFIABI Fl  (LOOP  CttACK) 

00147 

VALUE!  0,0 

00143 

rpon  in*  hk  i 

00144 

(DOOR  CBACK) 

00145 

RELATIVES; 

00146 

((OPTION  (DOOP 

C5ACK)  (C  HA  I BRACK  CBACK)  (WALL  CBACK))) 

00147 

( ( lip  ••  (OPTION 

(D'JOP  CBACK)  (CHAIRBACK  CBACK)  (wALL  CBACK)) 

00146 
00)  49 
00150 

[•HI  CBACK)  (CHA1PBACK  CBACK)  (DOOP  CHACK1)) 
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00154 

00155 

00156 

00157 
00159 
00159 
OOtoO 
00151 
0015? 

00153 

00154 
00165 
00165 
00167 
00189 
00169 
00171) 

00171 

00172 
0017  3 

00174 

00175 

00176 

00177 

00178 

00179 

00180 
00191 
0018? 
00  1 R 3 
001«4 
00195 
0018b 
00187 
00189 

00189 

00190 

00191 

00192 

00193 

00194 

00195 

00196 

00197 
00196 
001Q9 
00200 
00201 
00202 

00203 

00204 


l A ND  • .11  (OP**  (OPTION  ( DOOR  CBACK) 

(CH&IRBACK  CBACK  ) 

("ALL  CRACK)) 

(AND*#  (OP**  (OPTION  (CHAIR SEAT  CSEAT)  ) 
CCH4IRSEAT  CSEAT)) 

(OR**  (OPTION  (WASTEBASKET  W 0 S K T ) 
(DOOR  WBSKT) 

(WALL  WBSKT ) ) 
(WASTEBASKET  WBSKT) ) 

(OR**  (OPTION  (TABLETOP  TTOP)) 
(TABLETOP  TTOP) 

RELATIVES: 

((OPTION  (OOOP  CRACK ) (CHAJRBACK  CBACK)  (WALL  CBACK))) 
((OR.*  (OPTION  (LOOP  CBACK)  (CHAtPBACK  CBACK)  (WALL  CBACK)) 
(wALL  CBACK  ) (CrA  IRmACK  CBACK)  (DOOR  CBACK))) 


VARTAHLE:  (WALL.  CBACK) 

value:  o.o 

PROCEDURE : 

(KAI.L  CBACK) 

PFLATJVESf 

((OPTION  (DOOR  CBACK)  (CHA1RRACK  C HACK)  (WALL  CBACK))) 

( (OR**  (OPTION  (POOR  CBACK)  (CH6IRBACK  CBACK)  (WALL  CBACK)) 
(WALL  CBACK)  (CHAIRBACK  CBACK  ) (DOOR  CrACk))) 


VARIABLE:  (PICTURE  PTC) 

VALUE:  .3 

PROCEDURE  : 

(AND*  .3  (OR#*  (OPTION  (PICTURE  FIC) 

(DOOR  PIC) 

(wALL  PIC)) 

(AND#*  (OP**  (OPTION  (TAB1ETOP  TTOP)) 
(TABLETOP  TTOP)) 

(OP**  (OPTION  (DOOP  LWALL) 

(WAIL  LWALL)) 

(WALL  LWALL)) 

(OR**  (Option  (door  pwald 
(WALL  RWALL ) ) 

(WALL  RWALL1 

PE  LAI  IVES: 

((OPTION  (PICTURE  FIC)  (COOP  PIC)  (WALL  FIC))) 

((OP#*  (OPTION  (PICTURF  PIC)  (DOOR  PIC)  (WALL  PIC))  (DOOP 
PIC)  (PICTURE  RIC))) 


V API AHl E : (DOOR  PIC) 

VALUE : 0.0 

PROCEDURE  : 

(DOOR  PIC) 
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00205 

RELATIVES: 

00206 

((OPTION  (PICTURE  PIC)  (DOOR  PIC)  (WALL  PIC))) 

00207 

((OP**  (OPTION  (PICTURE  PIC)  (ODOR  PIC)  (WALL  PIC)) 

(DOOR 

002U8 

00209 

00210 

PIC)  (PICTURE  PIC))) 

0071  1 

VARIARLE;  (WALL  PIC) 

00212 

VALUE:  0.0 

0021  3 

PPOCEPt'RF: 

00214 

(WALL  PIC) 

00715 

PELATIVf.S: 

00216 

00217 

00218 

((UPTION  (PICTURE  PIC)  (DOOR  PIC)  (WALL  PIC))) 

00219 

V AH  ABLE  : (DOOR  OR) 

00220 

VALUE:  .227 

00221 

PROCEDURE : 

00222 

[AND*  .227  (DP**  (OPTION  (OnOF  DP) 

00223 

(WALL  DP)) 

00224 

(AND**  (OR**  (OPTION  (WASTEBASKET 

WBSKT) 

00225 

(DOOR  WBSKT) 

00226 

(WALL  WBSKTl) 

00227 

(WALL  WPSKT) 

00228 

(DOOR  WBSKT) 

00229 

(WASTEBASKET  WBSKT)) 

00230 

(OR**  (OPTION  (DOOR  IWALL) 

00231 

(WALL  LWALL) ) 

0023? 

( * ALL  LWALL)) 

002  33 

(OR**  (OPTION  ( FLOOR  FLR ) ) 

002)4 

(FLOOR  FLR) 

00735 

RELATIVES: 

002  36 

((OPTION  (DOOR  DR)  (WALL  OR))) 

00237 

(((|R**  (OPTION  (0O0P  DP)  (WALL  DR))  (WALL  DR)  (DOOR 

DR)  ) ) 

00238 

00239 

00240 

((OR**  (OPTION  (DOOR  DR)  (WALL  DR))  (DOOR  DP))) 

00241 

VARIABLE:  (WALL  DR) 

0 0242 

VALUE:  0.0 

00243 

FROCEDDRE: 

00244 

( A uD*  .773  (OR**  (OPTION  (DOOR  PR) 

00245 

( a AlL  DR)) 

00246 

(AND**  (OR**  (OPTION  (WASTEBASKET 

WBSKT) 

00247 

(DOOR  WBSKT) 

00248 

(WALL  *BSKT ) ) 

00249 

(WALL  WBSKT) 

00250 

(DOOR  WBSKT) 

00251 

(WASTEBASKET  WBSKT)) 

0025? 

(OR**  (OPTION  (DOOR  LWALL) 

00253 

(WALL  LWALL)) 

00254 

(DOOR  LWALL)) 

00255 

(OR**  (OPTION  (FLOOR  FLP ) ) 

FIGURE  F-4  DATABASE  FOLLOWING  SEARCH  (Continued) 
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00256  ( F LOOP  FLP J 

00257  FFLATIVES: 

00258  ((OPTION  (0008  DM  ( wALL  PHI)) 

00259  ((OH**  (OPTION  (DOOR  PR)  (4IU  DR))  (WALL  DP)  (HOOP  DR))) 

002b0 
002b  1 

00262  16078  ronses 

00263  179.  seconds 


Appendix  G 

SOLVING  CONSTRAINT  SATISFACTION  PUZZLES  WITH  A VISION  SYSTEM 

The  puzzle,  defined  in  Figure  G-l , exemplifies  a class  of  constraint 
satisfaction  problems  that  are  very  difficult  to  solve  by  conventional 
heuristic  search  methods.  These  problems  have  state  spaces  that  are  far 
too  large  to  search  by  exhaustive  enumeration  and  usually,  there  are  no 
obvious  heuristics  for  selecting  instantiations.  Moreover,  backtracking 
is  of  limited  utility  because  many  of  the  constraints  cannot  be  tested 
until  several  problem  variables  have  been  instantiated. 


A reasonable  state  space  representation  for  the  problem  defined  in 
Figure  G-l  consists  of  56  (approximately  15,000)  sextuples,  each  containing 
instantiations  of  the  variables  nationality,  house  position,  house  color, 
drink,  cigarette,  and  pet  (e.g.,  (ENGLISHMAN,  MIDDLE  HOUSE,  RED,  TEA,  KOOLS, 
and  DOG)).  The  problem  is  solved  by  finding  five  sextuples,  having  unique 
instantiations  for  each  variable,  that  satisfy  all  15  constraints. 

An  exhaustive  search  of  this  space,  choosing  sextuples  five  at  a time, 
requires  examination  of  5s  y 4s  x 36  x 2B  y 1B  = 2 .5  y 1010  sets  of  sextuples. 
(There  are  five  choices  of  house  position,  house  color,  pet,  drink,  and 
cigarette  available  in  the  sextuple  corresponding  to  the  first  nationality, 
leaving  four  choices  in  each  category  free  for  the  second  nationality, 


three  choices  each  for  the  third  nationality,  and  so  forth.)  Even  if  these 
sets  of  sextuples  could  be  tested  against  the  constraints  at  a rate  of 
100/second,  it  would  still  take  almost  eight  years  to  complete  the  exhaustive 
search . 

Fortunately,  astute  puzzle  solvers  have  discovered  that  the  require- 
ments for  search  can  be  sharply  reduced  or  even  eliminated  by  using  infor- 
mation in  the  constraints  directly  to  eliminate  inconsistent  elemerts  from 
the  ranges  of  problem  variables.30* 33  Constraints  1 and  5,  for  example, 
allow  immediate  deletion  from  the  original  set  of  problem  states  of  all 
sextuples  containing  the  nationality  Ukranian  and  a drink  other  than  tea, 
as  well  as  all  those  containing  the  drink  tea  and  a nationality  other  than 
Ukranian.  Constraints  4,  9,  6,  and  13,  respectively,  eliminate  additional 
sextuples  in  which  the  nationality  Ukranian  is  paired  with  the  green  house, 
the  middle  house,  the  house  to  the  right  of  the  irovy  house,  and  the  house 
where  Luckys  are  smoked.  The  elimination  process  has  a cumulative  effect 
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1.  There  are  five  houses,  each  of  a different  color  and  inhabited  by  men  of  different  nationalities,  with  different 
pets,  drinks,  and  cigarettes. 

2.  The  Englishman  lives  in  the  red  house. 

3.  The  Spaniard  owns  the  dog. 

4.  Coffee  is  drunk  in  the  green  house. 

5.  The  Ukranian  drinks  tea. 

6.  The  green  house  is  immediately  to  the  right  (your  right)  of  the  ivory  house. 

7.  The  Old  Gold  smoker  owns  snails. 

8.  Kools  are  smoked  in  the  yellow  house. 

9.  Milk  is  drunk  in  the  middle  house. 

10.  The  Norwegian  lives  in  the  first  house  on  the  left. 

11.  The  man  who  smokes  Chesterfields  lives  in  the  house  next  to  the  man  with  the  fox. 

• 12.  Kools  are  smoked  in  the  house  next  to  the  house  where  the  horse  is  kept. 

13.  The  Lucky  Strike  smoker  drinks  orange  juice. 

14.  The  Japanese  smokes  Parliaments. 

15  The  Norwegian  lives  next  to  the  blue  house. 

WHO  DRINKS  WATER?  WHO  OWNS  THE  ZEBRA? 

FIGURE  G1  A CONSTRAINT  SATISFACTION  PUZZLE 


since  deletions  can  combine  with  other  constraints  to  trigger  further 
deletions.  Thus,  from  constraint  6,  whenever  ivory  is  eliminated  as 
a possible  color  for  one  of  the  houses,  green  can  be  immediately  elimi- 
nated as  a possible  color  for  the  house  immediately  to  the  right. 

Moreover,  by  constraint  1,  eliminating  all  but  one  of  the  possible 
colors  for  one  house  allows  the  elimination  of  all  tuples  in  which 
that  color  is  paired  with  a different  house. 

The  constraint  satisfaction  mechanisms  in  MSYS  provide  an  ideal 
way  to  efficiently  propagate  such  deletions. 

A.  Solving  the  Problem  in  MSYS 

The  state  space  for  the  puzzle  in  Figure  G-l  was  represented  in 
MSYS  as  canonic  variables  representing  all  the  possible  pairwise  associ- 
ations between  different  problem  variables.  The  sextuple  (HI  English 
RED  KOOLS  TEA  DOG),  for  example,  was  partitioned  into  the  following 
15  MSYS  variables.* 

(HI  ENGLISH)  (HI  RED)  (HI  KOOLS)  (HI  TEA)  (HI  DOG) 

(ENGLISH  RED)  (ENGLISH  KOOLS)  (ENGLISH  TEA)  (ENGLISH  DOG) 

(RED  KOOLS)  (RED  TEA)  (RED  DOG) 

(KOOLS  TEA)  (KOOLS  DOG) 
(TEA  DOG) 

A total  of  375  tuples  were  required  to  represent  the  complete  state  space. 

An  initialization  procedure  was  written  to  generate  these  variables 
and  set  their  initial  likelihoods  to  1.0.  The  likelihood  procedure  as- 
sociated with  each  tuple  was  a conjunction  containing  other  tuples  and 
disjunctions  of  tuples  whose  validity  provided  support  for  applicable 
constraints.  Deletions  were  propagated  by  reevaluating  the  procedures 
of  tuples  in  which  a deleted  tuple  appeared. 

Sets  of  tuples  representing  alternative  associations  for  the  same 
problem  variables  are  again  linked  by  OPTION  variables.  Each  tuple 
appears  in  two  OPTION  variables,  one  for  each  element  of  the  pair.  For 
example,  the  tuple  (HI  RED)  is  linked  to  all  possible  colors  for  house 

*The  five  houses  in  left-right  order  are  designated  as  HI H5. 
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HI  by  the  variable  (OPTION  (HI  RED)  (HI  YELLOW) (HI  BLUE))  and  to  all 

possible  houses  associated  with  the  color  red,  by  the  variable  (OPTION 

(HI  RED)  (H2  RED) (H5  RED)).  OPTION  procedures  again  perform  a variety 

of  functions,  such  as  checking  for  contradictions  (where  all  alternatives 
have  been  eliminated)  and  updating  the  instantiation  queue.  They  also 
play  a crucial  role  in  propagating  deletions  resulting  from  the  elimination 
of  all  but  one  of  their  alternatives,  as  explained  below.  OPTION  variables 
evaluate  to  the  number  of  currently  valid  alternatives  they  encompass, 
initially  5. 

MSYS  proceeds  towards  a solution  by  deleting  all  tuples  (i.e.,  setting 
their  likelihood  to  0.0)  shown  to  be  inconsistent  with  the  given  constraints. 
Constraints  2 through  15  were  represented  in  MSYS  by  the  procedural  con- 
straints ASSERT,  RIGHTOF,  andNEXTTO,  as  shown  in  Figure  G-2.  ASSERT  oper- 
ates by  deleting  all  remaining  alternatives  in  both  OPTION  variables  as- 
sociated with  the  asserted  pair,  For  example,  ASSERT (UKRAN IAN  TEA)  deletes 
all  tuples  of  the  form  (UKRANIAN— , TEA)  where— iTEA  designates  any  beverage 
other  than  TEA,  as  well  as  all  those  of  the  form  (— i UKRANIAN  TEA). 

The  constraint  RIGHTOF(IVORY  GREEN)  (specific  arguments  are  used  for 
clarity)  states  that  a house  cannot  be  IVORY  if  the  house  immediately  to 
its  right  cannot  be  green.  Similarly,  a house  cannot  be  colored  GREEN 
unless  the  house  adjacent  on  its  left  can  be  colored  IVORY.  This  constraint 
operates  by  deleting  all  tuples  of  the  form  (HI  GREEN),  1 < I < 5 for  which 
(HI-1  IVORY)  is  already  eliminated,  as  well  as  all  tuples  of  the  form 
(HI  IVORY)  for  which(HI+l  GREEN)  is  false.  (HI  GREEN)  and  (H5  IVORY)  are 
always  eliminated.  Any  remaining,  mutually  supporting  tuples  of  the  form 
(HI  IVORY),  (HI+1  GREEN)  are  then  added  to  each  others  likelihood  procedures 
so  that  the  subsequent  deletion  of  either  one  will  automatically  trigger 
deletion  of  the  other. 

The  constraint  NEXTT0(K00LS  HORSE)  operates  in  a similar  fashion  by 
deleting  all  variables  of  the  form  (HI  KOOLS)  when  neither  (HI-1  HORSE) 
or  (Hl+1  HORSE)  are  still  valid.  Similarly,  all  variables  of  the  form 
(HI  HORSE)  unsupported  by  either  (Hl-1  KOOLS)  or  (HI+1  KOOLS)  get  deleted. 
Support  clauses  are  then  formulated  for  surviving  instances  of  (HI  KOOLS) 
and  (HI  HORSE)  and  added  to  their  likelihood  procedures.  These  clauses 
are  expressed  as  disjunctions  of  the  form  (OR  (HI-1  HORSE) (HI+1  HORSE)) 
and  (OR  (HI-1  KOOLS) (HI+1  KOOLS))  for  instances  of  KOOLS  and  HORSE, 
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(ASSERT  «°>  (ENGLISH  RED))) 
(ASSERT  (@  (SPANISH  DOG))) 
(ASSERT  (@  (GREEN  COFFEE))) 
(ASSERT  (@  (UKRAN  TEA))) 

(ASSERT  (@  (OLDGOLD  SNAILS))) 
(ASSERT  «9>  (YELLOW  KOOLS))) 
(ASSERT  «a>  (H3  MILK))) 

(ASSERT  (@  (HI  NORWEG))) 

(ASSERT  «s>  (LUCKYS  OJ)>) 

(ASSERT  (@  (JAPAN  PARLIAMENT))) 
(R1GHTOF  (IVORY  GREEN)) 

(NEXTTO  (CHESTERFIELDS  FOX)) 
NEXTTO  (KOOLS  HORSE)) 

(NEXTTO  (NORWEG  BLUE)) 


FIGURE  G2 


MSYS  REPRESENTATION  OF  CONSTRAINTS 


respectively.  The  disjunctions  prevent  deletion  of  the  supported  variable 
unless  the  required  supporting  interpretation  has  been  eliminated  at  both 
adjacent  houses. 

Relatively  few  variables  are  directly  eliminated  by  constraints  2 
through  15.  Far  more  inconsistencies  are  deduced  by  a process  of  elimina- 
tion based  on  the  uniqueness  requirements  expressed  in  constraint  1.  MSYS 
capitalizes  on  uniqueness  in  two  ways.  First,  whenever  an  OPTION  procedure 
observes  that  all  but  one  alternative  has  been  eliminated,  it  eliminates 
variables  associated  with  the  surviving  alternative  through  that  survivor's 
other  OPTION  variable.  For  example,  eliminating  (HI  YELLOW),  (HI  GREEN), 

(HI  IVORY),  and  (HI  BLUE)  causes  the  OPTION  variable  associated  with  the 
color  of  HI  to  delete  all  variables  that  are  inconsistent  with  the  surviving 
alternative  (HI  RED);  namely,  (H2  RED)  (H3  RED)  (H4  RED)  and  (H5  RED). 

Second,  a tuple  can  be  eliminated  when  both  of  its  components  are 
uniquely  associated  with  different  values  of  a third  variable.  Thus,  the 
assertion  of  (ENGLISHMAN  RED)  and  (COFFEE  GREEN)  requires  elimination  of 
(ENGLISHMAN  COFFEE).  The  mechanism  for  propagating  this  kind  of  deletion 
is  set  up  by  the  ASSERT  procedure.  A set  of  functions  are  created  that 
compare  pairs  of  OPTION  variables,  linking  the  two  elements  of  the  asser- 
tion with  alternative  bindings  of  a common  third  variable.  For  example, 
the  ASSERTION  ( ENGLISHMAN  RED)  creates  a function 

(MATCH  [OPTION  (COFFEE  ENGLISHMAN) (WATER  ENGLISHMAN)] 

[OPTION  (COFFEE  RED) (WATER  RED)]) 

for  comparing  the  OPTION  variables  associating  ENGLISHMAN  and  RED  with 
alternative  beverages.  Similar  expressions  are  created  linking  ENGLISHMAN 
and  RED  to  alternative  house  numbers,  pets,  and  cigarettes.  The  function 
MATCH  compares  corresponding  alternatives  (left  to  right)  in  a pair  of 
OPTION  variables  and  deletes  any  alternative  whose  opposite  number  is 
invalid.  Evaulation  of  this  function  is  automatically  triggered  whenever 
the  value  of  either  OPTION  variable  is  altered  by  deletion  of  one  of  its 
alternatives.  In  particular,  the  elimination  of  (COFFEE  RED)  following 
the  assertion  of  (COFFEE  GREEN)  triggers  execution  of  the  MATCH  variable 
illustrated  above,  resulting  in  the  elimination  of  (ENGLISHMAN  COFFEE). 


5 


rr 


G-6 


Execution  of  the  constraints  shown  in  Figure  G-2  and  propagation  of 
their  consequences,  immediately  eliminates  193  of  the  375  original  variables. 
Further  state-space  reductions  must  then  be  accomplished  by  instantiation. 

The  search  proceeds  as  in  scene  interpretation  by  popping  from  IQUEUE  the  OPTION 
variable  with  fewest  remaining  alternatives  and  then  instantiating  one  of 
those  alternatives.* 

The  system  was  fortunate  in  selecting  a correct  hypothesis  (UKRANIAN  BLUE) 

from  the  OPTION  variable  (OPTION  (NORWEGIAN  BLUE)  (UKRANIAN  BLUE) (JAPAN  BLUE)) 

for  its  first  instantiation.  Deductions  propagated  from  this  single  asser- 
tion led  directly  to  a successful  termination  consisting  of  the  75  surviving 
variables  shown  in  Figure  G-3.  These  variables  provide  unique  associations 
between  people,  beverages,  houses,  house  colors,  pets,  and  cigarettes 
summarized  in  Figure  G-4  that  satisfy  all  problem  constraints.  (The  reader 
might  wish  to  try  his  own  skill  before  peeking  at  the  nationalities  associated 
with  Water  and  Zebra.)  Altogether,  the  solution  required  almost  30  minutes 
of  PDP-10  CPU  time  running  in  interpreted  LISP.  About  one-half  of  this  time 
was  spent  in  setting  up  the  data  base  and  in  the  initial  phase  of  constraint 
satisfaction  prior  to  instantiation.  The  power  of  the  constraint  satisfaction 
approach  utilized  by  MSYS  compared  with  an  exhaustive  search  is 
clear. 

The  OPTION  variable  used  in  the  above  instantiation  was  one  of  35 
candidates  with  three  eliminated  alternatives,  all  of  equal  priority.  A 
less  fortuitous  instantiation  might  have  eliminated  fewer  variables  or 
even  led  to  a contradiction.  In  either  case,  MSYS  would  then  have  explored 
in  a separate  context  the  consequences  of  denying  the  hypothesis.  The 
search  would  then  have  continued,  if  necessary,  in  whichever  context  had 
the  largest  total  number  of  eliminated  variables.  A few  experiments  were 
performed  wherein  incorrect  hypotheses  were  purposefully  asserted  and 
correct  ones  denied.  These  errors  led  to  immediate  contradictions. 


*In  the  absence  of  real  valued  Boolean  likelihoods,  the  priority  of  OPTION 
variables  on  IQUEUE  was  based  on  the  number  of  their  alternatives  already 
eliminated.  OPTION  variables,  reduced  to  a unique  alternative,  were  re- 
moved from  the  queue.  The  instantiation  priority  of  alternatives  within 
an  OPTION  variable  was  assigned  arbitrarily  based  on  their  position  in 
the  OPTION  expression. 
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(COFFEE  ZEBRA)  (OJ  DOG)  (MILK  SNAILS)  (TEA  HORSE)  (WATER  FOX)  (PARLIAMENT 
ZEBRA)  (LUCKS  DOG)  (OLDGOLD  SNAILS)  (CHESTERFIELDS  HORSE)  (KOOLS  FOX) 
(PARLIAMENT  COFFEE)  (LUCKYS  OJ)  (OLDGOLD  MILK)  (CHESTERFIELDS  TEA)  (KOOLS 
WATER)  (GREEN  ZEBRA)  (IVORY  DOG)  (RED  SNAILS)  (BLUE  HORSE)  (YELLOW  FOX) 
(GREEN  COFFEE)  (IVORY  OJ)  (RED  MILK)  (BLUE  TEA)  (YELLOW  WATER)  (GREEN 
PARLIAMENT)  (IVORY  LUCKYS)  (RED  OLDGOLD)  (BLUE  CHESTERFIELDS)  (YELLOW 
KOOLS)  (JAPAN  ZEBRA)  (SPANISH  DOG)  (ENGLISH  SNAILS)  (UKRAN  HORSE)  (NORWEG 
FOX)  (JAPAN  COFFEE)  (SPANISH  OJ)  (ENGLISH  MILK)  (UKRAN  TEA)  (NORWEG  WATER) 
(JAPAN  PARLIAMENT)  (SPANISH  LUCKYS)  (ENGLISH  OLDGOLD)  (UKRAN  CHESTERFIELDS) 
(NORWEG  KOOLS)  (JAPAN  GREEN)  (SPANISH  IVORY)  (ENGLISH  RED)  (UKRAN  BLUEI 
(NORWEG  YELLOW)  (H5  ZEBRA)  (H4  DOG)  (H3  SNAILS)  (H2  HORSE)  (HI  FOX)  (H5  COFFEE) 
(H4  OJ)  (H3  MILK)  (H2  TEA)  (HI  WATER)  (H5  PARLIAMENT)  (H4  LUCKYS)  (H3  OLDGOLD) 
IH2  CHESTERFIELDS)  (HI  KOOLS)  (H5  GREEN)  (H4  IVORY)  IH3  RED)  IH2  BLUE) 

(HI  YELLOW)  (H5  JAPAN)  (H4  SPANISH)  (H3  ENGLISH)  (H2  UKRAN)  (HI  NORWEG) 

FIGURE  G-3  FINAL  STATE  OF  DATA  BASE 
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m 

H2 

H3 

H4 

Hj 

NATIONALITY 

NORWEGIAN 

UKRANIAN 

ENGLISH 

SPANISH 

JAPANESE 

COLOR 

YELLOW 

BLUE 

RED 

IVORY 

GREEN 

PET 

FOX 

HORSE 

SNAILS 

DOG 

ZEBRA 

CIGARETTE 

KOOLS 

CHESTERFIELD 

OLD  GOLD 

LUCKYS 

PARLIAMENT 

DRINK 

WATER 

TEA 

MILK 

OJ 

COFFEE 

FIGURE  G4  SUMMARY  OF  ASSOCIATIONS  BETWEEN  PROBLEM  VARIABLES 
PROVIDED  BY  VARIABLES  IN  FIGURE  G3 
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